-- Logs begin at Tue 2022-07-12 16:40:04 UTC, end at Tue 2022-07-12 18:02:04 UTC. -- Jul 12 17:18:25.396288 dv-cinder-848759-2-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Jul 12 17:18:25.406575 dv-cinder-848759-2-iscsi devstack@keystone.service[80438]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Jul 12 17:18:25.408667 dv-cinder-848759-2-iscsi devstack@keystone.service[80438]: *** Starting uWSGI 2.0.18-debian (64bit) on [Tue Jul 12 17:18:25 2022] *** Jul 12 17:18:25.408667 dv-cinder-848759-2-iscsi devstack@keystone.service[80438]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Jul 12 17:18:25.408667 dv-cinder-848759-2-iscsi devstack@keystone.service[80438]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Jul 12 17:18:25.408667 dv-cinder-848759-2-iscsi devstack@keystone.service[80438]: nodename: dv-cinder-848759-2-iscsi Jul 12 17:18:25.408667 dv-cinder-848759-2-iscsi devstack@keystone.service[80438]: machine: x86_64 Jul 12 17:18:25.408667 dv-cinder-848759-2-iscsi devstack@keystone.service[80438]: clock source: unix Jul 12 17:18:25.408667 dv-cinder-848759-2-iscsi devstack@keystone.service[80438]: pcre jit disabled Jul 12 17:18:25.408667 dv-cinder-848759-2-iscsi devstack@keystone.service[80438]: detected number of CPU cores: 4 Jul 12 17:18:25.408667 dv-cinder-848759-2-iscsi devstack@keystone.service[80438]: current working directory: / Jul 12 17:18:25.408667 dv-cinder-848759-2-iscsi devstack@keystone.service[80438]: detected binary path: /usr/bin/uwsgi-core Jul 12 17:18:25.408667 dv-cinder-848759-2-iscsi devstack@keystone.service[80438]: your processes number limit is 47680 Jul 12 17:18:25.408667 dv-cinder-848759-2-iscsi devstack@keystone.service[80438]: your memory page size is 4096 bytes Jul 12 17:18:25.409378 dv-cinder-848759-2-iscsi devstack@keystone.service[80438]: detected max file descriptor number: 2048 Jul 12 17:18:25.409378 dv-cinder-848759-2-iscsi devstack@keystone.service[80438]: lock engine: pthread robust mutexes Jul 12 17:18:25.409378 dv-cinder-848759-2-iscsi devstack@keystone.service[80438]: thunder lock: enabled Jul 12 17:18:25.409378 dv-cinder-848759-2-iscsi devstack@keystone.service[80438]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Jul 12 17:18:25.409378 dv-cinder-848759-2-iscsi devstack@keystone.service[80438]: Python version: 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0] Jul 12 17:18:25.454012 dv-cinder-848759-2-iscsi devstack@keystone.service[80438]: Python main interpreter initialized at 0x564b64f5a9c0 Jul 12 17:18:25.454012 dv-cinder-848759-2-iscsi devstack@keystone.service[80438]: python threads support enabled Jul 12 17:18:25.454012 dv-cinder-848759-2-iscsi devstack@keystone.service[80438]: your server socket listen backlog is limited to 100 connections Jul 12 17:18:25.454012 dv-cinder-848759-2-iscsi devstack@keystone.service[80438]: your mercy for graceful operations on workers is 90 seconds Jul 12 17:18:25.454416 dv-cinder-848759-2-iscsi devstack@keystone.service[80438]: mapped 403077 bytes (393 KB) for 2 cores Jul 12 17:18:25.454525 dv-cinder-848759-2-iscsi devstack@keystone.service[80438]: *** Operational MODE: preforking *** Jul 12 17:18:25.454786 dv-cinder-848759-2-iscsi devstack@keystone.service[80438]: *** uWSGI is running in multiple interpreter mode *** Jul 12 17:18:25.454786 dv-cinder-848759-2-iscsi devstack@keystone.service[80438]: spawned uWSGI master process (pid: 80438) Jul 12 17:18:25.455004 dv-cinder-848759-2-iscsi systemd[1]: Started Devstack devstack@keystone.service. Jul 12 17:18:25.455652 dv-cinder-848759-2-iscsi devstack@keystone.service[80438]: spawned uWSGI worker 1 (pid: 80451, cores: 1) Jul 12 17:18:25.458468 dv-cinder-848759-2-iscsi devstack@keystone.service[80438]: spawned uWSGI worker 2 (pid: 80452, cores: 1) Jul 12 17:18:25.458468 dv-cinder-848759-2-iscsi devstack@keystone.service[80438]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Jul 12 17:18:31.018751 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jul 12 17:18:31.018751 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: from cryptography.utils import int_from_bytes Jul 12 17:18:31.019577 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jul 12 17:18:31.019577 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: from cryptography.utils import int_from_bytes Jul 12 17:18:31.024133 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jul 12 17:18:31.024133 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: from cryptography.utils import int_from_bytes Jul 12 17:18:31.024738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jul 12 17:18:31.024738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: from cryptography.utils import int_from_bytes Jul 12 17:18:31.069240 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Jul 12 17:18:31.069428 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Jul 12 17:18:31.069634 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] command line args: [] {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Jul 12 17:18:31.069922 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Jul 12 17:18:31.070108 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ================================================================================ {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Jul 12 17:18:31.070369 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] admin_token = **** {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.070653 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] config_dir = [] {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.070848 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.071085 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] config_source = [] {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.071338 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] debug = True {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.071545 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.072775 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.072775 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] insecure_debug = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.072775 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.072775 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.072775 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] list_limit = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.073312 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] log_config_append = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.073427 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Jul 12 17:18:31.073508 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.073508 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] log_dir = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.073691 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Jul 12 17:18:31.073817 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] log_file = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.073907 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] command line args: [] {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Jul 12 17:18:31.073988 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.074105 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Jul 12 17:18:31.074181 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.074302 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ================================================================================ {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Jul 12 17:18:31.074385 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.074510 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] admin_token = **** {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.074594 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.074677 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] config_dir = [] {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.074795 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.074879 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.075008 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.075094 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] config_source = [] {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.075094 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] debug = True {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.075236 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.075311 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.075418 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.075501 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.075577 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.075650 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] insecure_debug = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.075718 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.075882 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.075996 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.076126 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.076210 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.076363 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] list_limit = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.076484 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.076582 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] log_config_append = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.076712 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] notification_format = cadf {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.076835 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.076951 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] log_dir = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.077070 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.077187 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] log_file = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.077315 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.077430 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.16/identity {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.077552 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.077674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] publish_errors = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.077787 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.077909 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.078025 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.078149 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.078272 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.078397 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.078622 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.078750 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.078868 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.078988 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.078988 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] standard_threads = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.079219 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.079348 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] strict_password_check = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.079470 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.079591 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.079711 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.079832 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] use_eventlog = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.079949 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.080072 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] use_journal = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.080072 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] use_json = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.080288 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.080409 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] use_stderr = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.080546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.080675 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] use_syslog = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.080791 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] notification_format = cadf {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.080912 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] watch_log_file = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.080912 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.081135 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.081264 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.081264 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.081489 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.16/identity {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.081489 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] publish_errors = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.081788 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.081913 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.082030 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.082146 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.082261 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.082383 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.082542 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.082684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.082808 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.082931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.083054 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.083177 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] standard_threads = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.083303 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.083434 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] strict_password_check = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.083558 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.083681 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.083797 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.083912 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] use_eventlog = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.083912 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] use_journal = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.084138 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.084254 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] use_json = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.084374 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.084498 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] use_stderr = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.084498 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] use_syslog = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.084733 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.084856 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] watch_log_file = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 12 17:18:31.085007 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.085128 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.085245 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.085366 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.085491 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.085491 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.085717 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.085838 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.085960 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.086082 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.086206 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.086338 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.086489 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.086620 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.086744 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.086861 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.086987 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.087111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.087234 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.087356 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.087479 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.087603 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.087795 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.087795 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.088014 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.088129 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.088250 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.088374 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.088491 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.088611 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.088735 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.088851 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.088972 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.089100 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.089217 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.089332 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.089454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.089570 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.089690 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.089814 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.089937 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.090054 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.090178 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.090294 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.090493 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.090493 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.090729 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.090845 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.090966 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.091089 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.091211 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.091321 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.091426 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.091526 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.091623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.091757 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.091867 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.091979 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.092080 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.092189 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.092287 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.092389 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.092513 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.092631 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.092754 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.092887 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.093047 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.093172 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.093296 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.093425 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.093543 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.093668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.093668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.093894 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.094125 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.094244 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.094374 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.094521 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.094640 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.094756 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.094871 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.094999 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.095116 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.095240 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.095357 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.095472 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.095588 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.095709 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.095838 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.095957 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.096073 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.096194 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.096303 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.096426 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.096556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.096680 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.096797 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.096919 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.097037 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.097153 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.097277 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.097401 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.097509 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.097624 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.097739 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.097855 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.097979 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.098104 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.098104 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.098321 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.098453 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.098583 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.098710 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.098828 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.098944 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.099066 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.099184 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.099306 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.099451 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.099561 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] cache.enabled = True {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.099679 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.099802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.099959 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.100076 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.100275 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.100402 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.100520 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.100644 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.100763 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.100886 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.100886 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.101128 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.101256 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.101380 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.101497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.101627 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.101746 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.101862 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.101987 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.102119 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.102241 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.102890 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.103007 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.103123 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.103252 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.103372 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.103495 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.103612 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.103728 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.103844 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.103968 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.104092 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.104216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.104339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] cache.enabled = True {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.104339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.104568 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.104686 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.104802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.104918 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.105033 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.105149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.105273 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.105512 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.105640 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.105640 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.105867 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.105992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.106118 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.106236 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.106353 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.106483 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.106601 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.106718 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.106842 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.106961 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.107084 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.107209 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] auth.external = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.107327 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.107443 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] auth.mapped = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.107571 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.107690 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.107808 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.107924 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.108040 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.108156 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] auth.password = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.108283 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.108408 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] auth.token = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.108532 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.108656 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.108774 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.108890 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] catalog.caching = True {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.109015 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.109134 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.109250 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.109366 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.109482 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.109598 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.109730 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.109876 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.110013 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.110133 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.110256 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.110375 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] credential.caching = True {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.110542 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.110662 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] credential.driver = sql {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.110796 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.111180 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.111334 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.111454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.111570 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.111687 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.111687 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.111917 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] auth.external = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.112034 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.112151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] auth.mapped = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.112267 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.112384 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.112511 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.112635 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.112754 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.112877 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] auth.password = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.112996 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.113112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] auth.token = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.113238 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.113357 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.113473 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.113596 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] catalog.caching = True {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.113715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.113833 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.113967 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.114086 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.114203 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.114319 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.114451 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] federation.caching = True {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.114570 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.114763 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.114892 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.115019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] federation.driver = sql {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.115138 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] credential.caching = True {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.115255 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.115377 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] credential.driver = sql {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.115504 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.115623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.115740 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.115865 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.115985 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.116100 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.116226 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.116344 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.116461 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.116577 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.116693 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.116810 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.116935 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.117053 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.117170 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.117286 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.117403 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] identity.caching = True {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.117527 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.117646 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.117762 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.117879 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.117879 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.118747 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.118881 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.119003 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.119128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] identity.driver = sql {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.119249 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.119367 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.119484 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.119610 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.119739 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] federation.caching = True {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.119858 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.120050 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.120169 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.120286 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] federation.driver = sql {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.120411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.120537 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.120656 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.120773 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.120890 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.121007 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.121133 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.121253 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.121371 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.121488 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.121605 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.121722 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.121848 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.121967 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.122086 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.122205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.122322 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.122482 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.122619 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.122740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] identity.caching = True {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.122842 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.122959 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.123076 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.123193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.123320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.123440 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.123558 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.123676 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.123794 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.123911 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] identity.driver = sql {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.124036 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.124156 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.124272 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.124390 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.124506 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.124623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.124748 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.124867 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.124984 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.125101 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.125217 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.125333 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.125458 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.125578 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.125694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.125820 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.125940 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.126056 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.126181 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.126306 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.126471 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.126471 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.password = **** {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.126696 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.126813 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.126943 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.127063 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.127181 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.127298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.127415 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.127533 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.127658 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.127776 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.127892 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.128008 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.128131 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.128249 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.128470 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.128601 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.128719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.128837 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.128953 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.129080 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.129199 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.129316 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.129433 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.129581 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.129704 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.129832 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.129952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.130069 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.130192 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.130311 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.user = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.130454 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.130587 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.130708 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.130825 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.130942 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.131059 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.131176 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.131301 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.131419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.131536 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.131654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.password = **** {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.131771 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.131888 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.132013 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.132132 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.132249 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.132365 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.132466 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.132582 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.132709 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.132828 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.132945 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.133061 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.133178 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.133294 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.133420 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.133551 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.133691 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.133810 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.133912 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.134036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.134199 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.134323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.134497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.134618 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.134735 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.134853 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.134983 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.135103 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.135221 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.135338 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.user = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.135454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.135571 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.135696 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.135815 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.135933 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.136051 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.136167 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.136284 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.136414 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.136534 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.136658 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.136776 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.136928 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] policy.driver = sql {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.137047 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.137173 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.137292 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.137408 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.137525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.137641 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.137758 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.137883 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] receipt.caching = True {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.138002 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.138119 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.138235 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.138351 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.138494 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.138690 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.138812 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.138929 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.139047 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.139164 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.139280 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.139405 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] resource.caching = True {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.139524 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.139644 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.139762 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.139878 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] resource.driver = sql {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.139995 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.140121 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.140240 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.140357 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.140473 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.140589 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.140706 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.140830 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] revoke.caching = True {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.140949 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.141067 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.141183 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.141300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.141416 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.141542 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] role.cache_time = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.141542 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] role.caching = True {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.141763 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.141880 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] role.driver = sql {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.141996 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] policy.driver = sql {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.142112 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] role.list_limit = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.142238 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.142356 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.142485 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.142603 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.142720 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.142850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.142970 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] receipt.caching = True {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.143086 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.143204 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.143320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.143436 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.143562 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.143681 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.143798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.143915 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.144032 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.144149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.144275 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.144394 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] resource.caching = True {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.144517 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.144635 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.144752 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.144869 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] resource.driver = sql {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.144994 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.145113 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.145229 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.145375 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.145375 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.145598 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.145723 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] revoke.caching = True {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.145841 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.145959 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.146075 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.146192 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.146316 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.146477 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] role.cache_time = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.146598 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.146716 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] role.caching = True {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.146832 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.146949 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] role.driver = sql {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.147066 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.147195 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] role.list_limit = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.147315 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.147433 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.147550 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.147667 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.147784 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.147909 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.148028 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.148145 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.148262 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.148379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.148498 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.148623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.148742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.148858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.148975 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.149098 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.149216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.149342 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.149461 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.149583 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.149701 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.149826 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.149953 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] token.cache_time = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.150072 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.150190 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] token.caching = True {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.150307 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.150442 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.150442 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] token.provider = fernet {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.150682 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.150802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.150919 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.151035 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.151152 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.151269 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.151394 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.151512 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.151630 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.151746 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.151863 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.151980 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.152105 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.152223 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] trust.driver = sql {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.152340 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.152457 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.152457 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.152677 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.152802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.152921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.153037 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.153155 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.153271 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.153387 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.153513 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.153632 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.153750 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.153866 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.153982 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.154099 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.154223 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] database.connection = **** {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.154358 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.154503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.154624 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.154742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.154859 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.154975 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.155100 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] token.cache_time = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.155219 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.155336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] token.caching = True {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.155453 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.155570 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.155686 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.155812 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] token.provider = fernet {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.155931 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.156048 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.156165 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.156281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.156399 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.156523 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.156642 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.156759 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.156877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.157093 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.157222 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.157352 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.157472 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.157598 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] trust.driver = sql {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.157720 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.157838 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.157963 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.158090 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.158209 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.158327 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.158497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.158617 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.158735 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.158735 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80452) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Jul 12 17:18:31.158972 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.158972 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.158972 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.158972 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.158972 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] database.connection = **** {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.158972 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.158972 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.158972 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.158972 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.160017 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.160017 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.160017 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.160017 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.160448 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.160448 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.160668 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.160858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.161116 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.161361 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.161616 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.161854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.162103 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.162347 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 12 17:18:31.162548 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80451) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Jul 12 17:18:31.188819 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=80452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.189050 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=80452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.189285 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=80452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.190650 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=80451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.190892 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=80451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.191127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=80451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.198256 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=80452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.199789 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=80451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.226784 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=80452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.227015 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=80452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.229548 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=80451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.229785 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=80451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.374087 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=80452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.375513 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=80451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.424179 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=80452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.424411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=80452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.425830 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=80451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.426072 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=80451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.490447 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.490699 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=80452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.490927 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=80452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.491153 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=80452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.491373 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=80452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.491589 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=80452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.491675 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.491842 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=80451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.492076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=80451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.492308 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=80451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.492534 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=80451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.492758 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=80451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.518105 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.518332 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=80452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.518589 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=80452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.518678 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.518821 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=80452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.518907 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=80451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.519048 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=80451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.519159 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=80452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.519268 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=80452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.519362 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=80451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.519486 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=80451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.519569 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.519706 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=80451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.519957 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.530840 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=80452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.531647 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=80451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.543581 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.543820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=80452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.544043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=80452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.544251 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.544341 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=80452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.544475 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=80452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.544559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=80451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.544694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=80452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.544781 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=80451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.544913 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80452) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.544997 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=80451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.545135 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=80451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.545355 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=80451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.545593 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80451) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 12 17:18:31.562444 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.562661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jul 12 17:18:31.563040 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.563461 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jul 12 17:18:31.563738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.563831 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.563964 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jul 12 17:18:31.564052 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.564254 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.564355 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.564514 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.564754 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jul 12 17:18:31.565534 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.565534 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.565701 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.565890 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.576314 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.576611 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.576870 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.577582 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.577978 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.578276 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.578607 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.578927 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.585439 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `MethodViewType` (API Prefix: ) [/credentials, /credentials/] {{(pid=80452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 12 17:18:31.587417 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `MethodViewType` (API Prefix: ) [/credentials, /credentials/] {{(pid=80451) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 12 17:18:31.589160 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `MethodViewType` (API Prefix: ) [/domains, /domains/] {{(pid=80452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 12 17:18:31.589443 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.589729 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.590012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.590299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.590589 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.590865 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.590994 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `MethodViewType` (API Prefix: ) [/domains, /domains/] {{(pid=80451) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 12 17:18:31.591127 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.591271 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.591401 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.591533 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.591675 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.591807 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.591939 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.592078 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.592227 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.592480 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.592756 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.593030 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.593305 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.593651 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.619768 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.621399 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `MethodViewType` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=80452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 12 17:18:31.621688 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.621892 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.623571 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `MethodViewType` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=80451) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 12 17:18:31.623857 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.627035 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `MethodViewType` (API Prefix: ) [/groups, /groups/] {{(pid=80452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 12 17:18:31.627271 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.627555 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.629377 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `MethodViewType` (API Prefix: ) [/groups, /groups/] {{(pid=80451) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 12 17:18:31.629657 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.629975 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.635300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `MethodViewType` (API Prefix: ) [/limits, /limits/] {{(pid=80452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 12 17:18:31.635594 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.637828 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `MethodViewType` (API Prefix: ) [/limits, /limits/] {{(pid=80451) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 12 17:18:31.638123 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.640104 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `MethodViewType` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=80452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 12 17:18:31.640391 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.640676 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.640960 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.641237 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.641516 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.641792 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.642062 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.642811 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `MethodViewType` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=80451) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 12 17:18:31.643106 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.643396 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.643687 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.643970 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.644250 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.644524 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.644796 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.661554 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.661840 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.665938 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.666222 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.667125 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=80452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 12 17:18:31.670760 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.670888 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=80451) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 12 17:18:31.671058 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.674554 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.674881 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.676176 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `MethodViewType` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=80452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 12 17:18:31.679806 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=80452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 12 17:18:31.680110 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `MethodViewType` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=80451) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 12 17:18:31.683606 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.683906 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.684204 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.684498 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.684627 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=80451) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 12 17:18:31.684788 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.685070 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.688496 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.688808 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.689098 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.689406 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.689695 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.689979 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.705363 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `MethodViewType` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=80452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 12 17:18:31.705655 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.705928 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.706194 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.709621 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `MethodViewType` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=80451) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 12 17:18:31.709911 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.710188 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.710484 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.714686 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.716393 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.716674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.719110 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.719979 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `MethodViewType` (API Prefix: ) [/policies, /policies/] {{(pid=80452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 12 17:18:31.720267 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.720552 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.720826 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.720988 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.721109 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.721203 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.724484 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `MethodViewType` (API Prefix: ) [/policies, /policies/] {{(pid=80451) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 12 17:18:31.724771 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.725058 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.725344 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.725631 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.734675 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `MethodViewType` (API Prefix: ) [/projects, /projects/] {{(pid=80452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 12 17:18:31.734950 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.735234 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.735517 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.736063 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.736356 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.736652 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.739527 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `MethodViewType` (API Prefix: ) [/projects, /projects/] {{(pid=80451) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 12 17:18:31.739886 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.740183 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.740476 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.740757 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.741045 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.741322 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.756585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `MethodViewType` (API Prefix: ) [/regions, /regions/] {{(pid=80452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 12 17:18:31.760117 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `MethodViewType` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=80452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 12 17:18:31.761766 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `MethodViewType` (API Prefix: ) [/regions, /regions/] {{(pid=80451) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 12 17:18:31.763941 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.765551 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.765646 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `MethodViewType` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=80451) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 12 17:18:31.767352 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `MethodViewType` (API Prefix: ) [/roles, /roles/] {{(pid=80452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 12 17:18:31.767633 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.767926 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.769014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.770656 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.772449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `MethodViewType` (API Prefix: ) [/roles, /roles/] {{(pid=80451) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 12 17:18:31.772730 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.773016 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.775583 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.777183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `MethodViewType` (API Prefix: ) [/services, /services/] {{(pid=80452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 12 17:18:31.780445 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.780544 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.780744 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.781034 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.781320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.782212 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `MethodViewType` (API Prefix: ) [/services, /services/] {{(pid=80451) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 12 17:18:31.785455 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.785826 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.786125 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.786401 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.791146 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `MethodViewType` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=80452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 12 17:18:31.791445 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.791727 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.796325 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `MethodViewType` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=80451) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 12 17:18:31.796617 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.796935 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.799591 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `MethodViewType` (API Prefix: ) [/users, /users/] {{(pid=80452) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 12 17:18:31.799875 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.800159 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.800438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.800717 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.800995 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.801275 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.801541 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.801922 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.802203 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.802499 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.802769 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.803047 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.803322 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=80452) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.804954 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `MethodViewType` (API Prefix: ) [/users, /users/] {{(pid=80451) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 12 17:18:31.805238 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.807586 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.807896 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.808216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.808534 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.808839 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.809122 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.809513 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.809831 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.810121 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.810395 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.810702 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.810986 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=80451) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 12 17:18:31.843282 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: WSGI app 0 (mountpoint='') ready in 6 seconds on interpreter 0x564b64f5a9c0 pid: 80452 (default app) Jul 12 17:18:31.846832 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0932b0f1-ee34-4b10-9c40-7877577b7491 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:31.847128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0932b0f1-ee34-4b10-9c40-7877577b7491 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:31.847418 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0932b0f1-ee34-4b10-9c40-7877577b7491 None None] PATH_INFO: `/v3` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:31.851194 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: WSGI app 0 (mountpoint='') ready in 6 seconds on interpreter 0x564b64f5a9c0 pid: 80451 (default app) Jul 12 17:18:31.851647 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1/1] 10.222.0.16 () {50 vars in 791 bytes} [Tue Jul 12 17:18:31 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:18:40.794317 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fdd494-2b12-4def-8b43-be1ee6423c7c None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:40.794919 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fdd494-2b12-4def-8b43-be1ee6423c7c None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:40.794919 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58fdd494-2b12-4def-8b43-be1ee6423c7c None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:40.799685 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1/2] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:18:40 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:18:40.805661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc7a23d-4a42-41a9-816d-8c8d6194e3fa None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:40.806164 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc7a23d-4a42-41a9-816d-8c8d6194e3fa None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:40.806277 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc7a23d-4a42-41a9-816d-8c8d6194e3fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:40.878167 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: INFO dbcounter [None req-2fc7a23d-4a42-41a9-816d-8c8d6194e3fa None None] Registered counter for database keystone Jul 12 17:18:40.901003 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG oslo_db.sqlalchemy.engines [None req-2fc7a23d-4a42-41a9-816d-8c8d6194e3fa None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=80452) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Jul 12 17:18:40.907559 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writer thread running {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Jul 12 17:18:41.088677 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG passlib.handlers.bcrypt [None req-2fc7a23d-4a42-41a9-816d-8c8d6194e3fa None None] detected 'bcrypt' backend, version '3.2.2' {{(pid=80452) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Jul 12 17:18:41.089105 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG passlib.handlers.bcrypt [None req-2fc7a23d-4a42-41a9-816d-8c8d6194e3fa None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=80452) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Jul 12 17:18:41.144647 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-2fc7a23d-4a42-41a9-816d-8c8d6194e3fa None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:18:41.146197 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-2fc7a23d-4a42-41a9-816d-8c8d6194e3fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:18:41.275027 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 2/3] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:18:40 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 471 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Jul 12 17:18:41.497969 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58fdd494-2b12-4def-8b43-be1ee6423c7c None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:18:41.573462 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: INFO dbcounter [None req-58fdd494-2b12-4def-8b43-be1ee6423c7c None None] Registered counter for database keystone Jul 12 17:18:41.593508 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG oslo_db.sqlalchemy.engines [None req-58fdd494-2b12-4def-8b43-be1ee6423c7c None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=80451) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Jul 12 17:18:41.600005 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writer thread running {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Jul 12 17:18:41.725260 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58fdd494-2b12-4def-8b43-be1ee6423c7c None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:18:41.730204 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f127fb-21fb-4f90-a95f-0acb53871a85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:18:41.730923 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f127fb-21fb-4f90-a95f-0acb53871a85 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:41.731215 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f127fb-21fb-4f90-a95f-0acb53871a85 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:41.731497 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f127fb-21fb-4f90-a95f-0acb53871a85 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:41.740963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f127fb-21fb-4f90-a95f-0acb53871a85 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:18:41.794705 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: WARNING keystone.common.rbac_enforcer.enforcer [None req-b3f127fb-21fb-4f90-a95f-0acb53871a85 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Jul 12 17:18:41.795346 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f127fb-21fb-4f90-a95f-0acb53871a85 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:18:41.806359 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 2/4] 10.222.0.16 () {58 vars in 1324 bytes} [Tue Jul 12 17:18:41 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 12 17:18:41.812628 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc7a23d-4a42-41a9-816d-8c8d6194e3fa None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:18:41.828323 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc7a23d-4a42-41a9-816d-8c8d6194e3fa None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:18:41.833041 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731920f9-7081-4d4b-806d-ed6e84e6b2cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:18:41.834246 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731920f9-7081-4d4b-806d-ed6e84e6b2cb admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:41.834573 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731920f9-7081-4d4b-806d-ed6e84e6b2cb admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:41.834860 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731920f9-7081-4d4b-806d-ed6e84e6b2cb admin admin] PATH_INFO: `/v3/projects/1d1f69915b004b66af4c48b7c59a15d8` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:41.840195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731920f9-7081-4d4b-806d-ed6e84e6b2cb admin admin] RBAC: Authorizing `identity:get_project(project_id=1d1f69915b004b66af4c48b7c59a15d8)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:18:41.892297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: WARNING keystone.common.rbac_enforcer.enforcer [None req-731920f9-7081-4d4b-806d-ed6e84e6b2cb admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Jul 12 17:18:41.892644 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731920f9-7081-4d4b-806d-ed6e84e6b2cb admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:18:41.898330 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 3/5] 10.222.0.16 () {56 vars in 1207 bytes} [Tue Jul 12 17:18:41 2022] GET /identity/v3/projects/1d1f69915b004b66af4c48b7c59a15d8 => generated 344 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:18:43.946960 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf6ec4e-d226-4fb2-8b02-b8b07f25ff6c None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:43.947481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf6ec4e-d226-4fb2-8b02-b8b07f25ff6c None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:43.947481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf6ec4e-d226-4fb2-8b02-b8b07f25ff6c None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:43.951664 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 3/6] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:18:43 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 12 17:18:43.957281 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405e87b9-714e-45a3-9e4b-259743cce9e1 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:43.957675 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405e87b9-714e-45a3-9e4b-259743cce9e1 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:43.957854 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405e87b9-714e-45a3-9e4b-259743cce9e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:43.977592 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-405e87b9-714e-45a3-9e4b-259743cce9e1 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:18:43.979307 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-405e87b9-714e-45a3-9e4b-259743cce9e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:18:43.987634 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 4/7] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:18:43 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Jul 12 17:18:44.010594 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf6ec4e-d226-4fb2-8b02-b8b07f25ff6c None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:18:44.022850 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf6ec4e-d226-4fb2-8b02-b8b07f25ff6c None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:18:44.027169 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641a4c02-6af4-4337-a24f-54dab948a0b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:18:44.027773 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641a4c02-6af4-4337-a24f-54dab948a0b1 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:44.028065 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641a4c02-6af4-4337-a24f-54dab948a0b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:44.028346 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641a4c02-6af4-4337-a24f-54dab948a0b1 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:44.036328 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641a4c02-6af4-4337-a24f-54dab948a0b1 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:18:44.038130 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641a4c02-6af4-4337-a24f-54dab948a0b1 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:18:44.047589 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 4/8] 10.222.0.16 () {58 vars in 1324 bytes} [Tue Jul 12 17:18:44 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 12 17:18:44.053202 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405e87b9-714e-45a3-9e4b-259743cce9e1 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:18:44.063107 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405e87b9-714e-45a3-9e4b-259743cce9e1 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:18:44.067644 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59d58f71-5a2b-400d-8897-8a5f3d07432b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:18:44.068513 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d58f71-5a2b-400d-8897-8a5f3d07432b admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:44.068801 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d58f71-5a2b-400d-8897-8a5f3d07432b admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:44.069081 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d58f71-5a2b-400d-8897-8a5f3d07432b admin admin] PATH_INFO: `/v3/users/df85f597d0ba471381a5a8f9b6886705` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:44.070704 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59d58f71-5a2b-400d-8897-8a5f3d07432b admin admin] RBAC: Authorizing `identity:get_user(user_id=df85f597d0ba471381a5a8f9b6886705)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:18:44.072596 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59d58f71-5a2b-400d-8897-8a5f3d07432b admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:18:44.074187 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 5/9] 10.222.0.16 () {56 vars in 1198 bytes} [Tue Jul 12 17:18:44 2022] GET /identity/v3/users/df85f597d0ba471381a5a8f9b6886705 => generated 247 bytes in 21 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:18:46.190021 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea40e5e-ba59-4597-85e5-62930230dc6f None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:46.190563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea40e5e-ba59-4597-85e5-62930230dc6f None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:46.190645 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea40e5e-ba59-4597-85e5-62930230dc6f None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:46.191987 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 5/10] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:18:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:18:46.197818 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae1b177-fea5-4ca2-95e4-a063b537fbc5 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:46.198258 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae1b177-fea5-4ca2-95e4-a063b537fbc5 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:46.198526 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae1b177-fea5-4ca2-95e4-a063b537fbc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:46.216290 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-4ae1b177-fea5-4ca2-95e4-a063b537fbc5 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:18:46.217471 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-4ae1b177-fea5-4ca2-95e4-a063b537fbc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:18:46.255003 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1206e69e-e339-481d-9e32-c284571aaa0c None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:46.255300 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1206e69e-e339-481d-9e32-c284571aaa0c None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:46.255588 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1206e69e-e339-481d-9e32-c284571aaa0c None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:46.256961 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 6/11] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:18:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 12 17:18:46.262238 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91891dc-f2c0-4351-9ff6-94a28e8b26e5 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:46.262570 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91891dc-f2c0-4351-9ff6-94a28e8b26e5 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:46.262863 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91891dc-f2c0-4351-9ff6-94a28e8b26e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:46.302850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 6/12] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:18:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:18:46.303992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG passlib.handlers.bcrypt [None req-f91891dc-f2c0-4351-9ff6-94a28e8b26e5 None None] detected 'bcrypt' backend, version '3.2.2' {{(pid=80451) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Jul 12 17:18:46.305153 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG passlib.handlers.bcrypt [None req-f91891dc-f2c0-4351-9ff6-94a28e8b26e5 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=80451) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Jul 12 17:18:46.327151 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae1b177-fea5-4ca2-95e4-a063b537fbc5 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:18:46.350278 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-f91891dc-f2c0-4351-9ff6-94a28e8b26e5 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:18:46.350868 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:18:46.352500 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-f91891dc-f2c0-4351-9ff6-94a28e8b26e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:18:46.438067 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ae1b177-fea5-4ca2-95e4-a063b537fbc5 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:18:46.459106 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 7/13] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:18:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 12 17:18:46.475958 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08339e07-1c80-4567-9d9c-4c2d2055c854 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:18:46.477280 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08339e07-1c80-4567-9d9c-4c2d2055c854 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:46.477563 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08339e07-1c80-4567-9d9c-4c2d2055c854 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:46.477838 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08339e07-1c80-4567-9d9c-4c2d2055c854 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:46.482641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91891dc-f2c0-4351-9ff6-94a28e8b26e5 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:18:46.489725 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08339e07-1c80-4567-9d9c-4c2d2055c854 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:18:46.501482 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-08339e07-1c80-4567-9d9c-4c2d2055c854 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:18:46.501482 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:18:46.501482 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:18:46.501482 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:18:46.501482 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:18:46.501482 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:18:46.501482 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:18:46.501482 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:18:46.501482 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:18:46.501482 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:18:46.501482 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:18:46.501482 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:18:46.501482 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:18:46.501482 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:18:46.501482 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:18:46.501482 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:18:46.502386 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:18:46.502386 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:18:46.502386 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:18:46.502386 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:18:46.502386 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:18:46.502386 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:18:46.502386 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:18:46.502386 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:18:46.502386 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:18:46.502386 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:18:46.502386 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:18:46.502386 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:18:46.502386 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:18:46.502386 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:18:46.502386 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:18:46.502386 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:18:46.503441 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:18:46.503441 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:18:46.503441 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:18:46.503441 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:18:46.503441 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:18:46.503441 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:18:46.503441 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:18:46.503441 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:18:46.503441 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:18:46.503441 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:18:46.503441 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:18:46.503441 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:18:46.503441 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 7/14] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Jul 12 17:18:46 2022] GET /identity/v3/roles/admin => generated 83 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:18:46.509641 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08339e07-1c80-4567-9d9c-4c2d2055c854 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:18:46.589852 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91891dc-f2c0-4351-9ff6-94a28e8b26e5 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:18:46.592219 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:18:46.596075 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:18:46.614856 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08339e07-1c80-4567-9d9c-4c2d2055c854 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:18:46.629618 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7294b60b-eaea-4ba7-95a1-6e7119468ffd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:18:46.630818 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7294b60b-eaea-4ba7-95a1-6e7119468ffd None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:46.631107 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7294b60b-eaea-4ba7-95a1-6e7119468ffd None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:46.631413 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7294b60b-eaea-4ba7-95a1-6e7119468ffd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:46.651031 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f12ef3-5563-43b9-9187-7db534b80c69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:18:46.651714 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f12ef3-5563-43b9-9187-7db534b80c69 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:46.652012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f12ef3-5563-43b9-9187-7db534b80c69 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:46.652344 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f12ef3-5563-43b9-9187-7db534b80c69 None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:46.653117 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f12ef3-5563-43b9-9187-7db534b80c69 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:18:46.653428 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f12ef3-5563-43b9-9187-7db534b80c69 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:18:46.654879 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f12ef3-5563-43b9-9187-7db534b80c69 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:18:46.667899 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 8/15] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:18:46 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:18:46.673445 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f12ef3-5563-43b9-9187-7db534b80c69 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:18:46.704899 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7294b60b-eaea-4ba7-95a1-6e7119468ffd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:18:46.706568 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7294b60b-eaea-4ba7-95a1-6e7119468ffd None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:18:46.774885 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f12ef3-5563-43b9-9187-7db534b80c69 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:18:46.808957 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5903a6-670b-41c1-81b6-c01c00aafd6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:18:46.809286 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 8/16] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:18:46 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 12 17:18:46.809838 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5903a6-670b-41c1-81b6-c01c00aafd6e None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:46.810121 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5903a6-670b-41c1-81b6-c01c00aafd6e None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:46.810398 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5903a6-670b-41c1-81b6-c01c00aafd6e None admin] PATH_INFO: `/v3/users/df85f597d0ba471381a5a8f9b6886705` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:46.811709 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5903a6-670b-41c1-81b6-c01c00aafd6e None admin] RBAC: Authorizing `identity:get_user(user_id=df85f597d0ba471381a5a8f9b6886705)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:18:46.813167 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5903a6-670b-41c1-81b6-c01c00aafd6e None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:18:46.818869 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 9/17] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:18:46 2022] GET /identity/v3/users/df85f597d0ba471381a5a8f9b6886705 => generated 247 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:18:46.819030 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7294b60b-eaea-4ba7-95a1-6e7119468ffd None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:18:46.819030 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:18:46.820388 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5903a6-670b-41c1-81b6-c01c00aafd6e None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:18:46.834885 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:18:46.917251 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7294b60b-eaea-4ba7-95a1-6e7119468ffd None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:18:46.921505 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5903a6-670b-41c1-81b6-c01c00aafd6e None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:18:46.952442 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7150ada1-2363-49f1-96ef-515b7ab67312 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:18:46.953242 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7150ada1-2363-49f1-96ef-515b7ab67312 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:46.953539 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7150ada1-2363-49f1-96ef-515b7ab67312 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:46.953822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7150ada1-2363-49f1-96ef-515b7ab67312 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:46.957471 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ad967e-453f-4b25-bde8-446a51aeebf9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:18:46.958269 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ad967e-453f-4b25-bde8-446a51aeebf9 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:46.958589 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ad967e-453f-4b25-bde8-446a51aeebf9 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:46.958876 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ad967e-453f-4b25-bde8-446a51aeebf9 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:46.960055 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ad967e-453f-4b25-bde8-446a51aeebf9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:18:46.962113 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02ad967e-453f-4b25-bde8-446a51aeebf9 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:18:46.963982 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 10/18] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:18:46 2022] GET /identity/v3/domains/default => generated 199 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:18:46.969550 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ad967e-453f-4b25-bde8-446a51aeebf9 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:18:46.975097 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7150ada1-2363-49f1-96ef-515b7ab67312 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:18:46.976617 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7150ada1-2363-49f1-96ef-515b7ab67312 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:18:46.999112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-7150ada1-2363-49f1-96ef-515b7ab67312 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:18:46.999112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:18:46.999112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:18:46.999112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:18:46.999112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:18:46.999112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:18:46.999112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:18:46.999112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:18:46.999112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:18:46.999112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:18:46.999112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:18:46.999112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:18:46.999112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:18:46.999112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:18:46.999112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:18:46.999970 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:18:46.999970 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:18:46.999970 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:18:46.999970 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:18:46.999970 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:18:46.999970 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:18:46.999970 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:18:46.999970 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:18:46.999970 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:18:46.999970 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:18:46.999970 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:18:46.999970 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:18:46.999970 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:18:46.999970 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:18:46.999970 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:18:46.999970 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:18:46.999970 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:18:47.000779 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:18:47.000779 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:18:47.000779 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:18:47.000779 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:18:47.000779 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 9/19] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:18:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:18:47.006231 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7150ada1-2363-49f1-96ef-515b7ab67312 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:18:47.052891 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:18:47.068095 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:18:47.074459 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ad967e-453f-4b25-bde8-446a51aeebf9 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:18:47.113144 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9e3c80-0bf6-407d-ad50-d7c5074890a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:18:47.113799 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9e3c80-0bf6-407d-ad50-d7c5074890a8 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:47.113927 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7150ada1-2363-49f1-96ef-515b7ab67312 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:18:47.114088 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9e3c80-0bf6-407d-ad50-d7c5074890a8 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:47.114363 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9e3c80-0bf6-407d-ad50-d7c5074890a8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:47.115230 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9e3c80-0bf6-407d-ad50-d7c5074890a8 None admin] RBAC: Adding query filter params (role.id=eb05bf286c894ea9bf44a6c023a4323b, scope.domain.id=default, user.id=df85f597d0ba471381a5a8f9b6886705) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:18:47.115544 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9e3c80-0bf6-407d-ad50-d7c5074890a8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:18:47.117416 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9e3c80-0bf6-407d-ad50-d7c5074890a8 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:18:47.124486 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 11/20] 10.222.0.16 () {56 vars in 1322 bytes} [Tue Jul 12 17:18:46 2022] GET /identity/v3/role_assignments?user.id=df85f597d0ba471381a5a8f9b6886705&scope.domain.id=default&role.id=eb05bf286c894ea9bf44a6c023a4323b => generated 233 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:18:47.152701 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0abe6c7-41f2-427e-8d6c-5e497adff245 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:18:47.153388 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0abe6c7-41f2-427e-8d6c-5e497adff245 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:47.153672 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0abe6c7-41f2-427e-8d6c-5e497adff245 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:47.153946 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0abe6c7-41f2-427e-8d6c-5e497adff245 None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:47.154725 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0abe6c7-41f2-427e-8d6c-5e497adff245 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:18:47.155040 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0abe6c7-41f2-427e-8d6c-5e497adff245 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:18:47.156580 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0abe6c7-41f2-427e-8d6c-5e497adff245 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:18:47.176326 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 10/21] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:18:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:18:49.649507 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e0cf81-345e-418c-bd0b-c762badbf9b0 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:49.650018 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e0cf81-345e-418c-bd0b-c762badbf9b0 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:49.650091 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e0cf81-345e-418c-bd0b-c762badbf9b0 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:49.651689 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 12/22] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:18:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:18:49.664779 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e437c8c3-3442-474c-838a-a447071f6aa0 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:49.665277 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e437c8c3-3442-474c-838a-a447071f6aa0 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:49.665381 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e437c8c3-3442-474c-838a-a447071f6aa0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:49.691909 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-e437c8c3-3442-474c-838a-a447071f6aa0 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:18:49.693137 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-e437c8c3-3442-474c-838a-a447071f6aa0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:18:49.742081 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aca3260-69c3-4238-b424-874fcddc8bbb None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:49.742934 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aca3260-69c3-4238-b424-874fcddc8bbb None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:49.743441 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aca3260-69c3-4238-b424-874fcddc8bbb None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:49.745017 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 13/23] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:18:49 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:18:49.750062 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec9ce5f-2938-4ed0-a7d7-a373689b759f None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:49.750612 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec9ce5f-2938-4ed0-a7d7-a373689b759f None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:49.751092 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec9ce5f-2938-4ed0-a7d7-a373689b759f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:49.787405 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-0ec9ce5f-2938-4ed0-a7d7-a373689b759f None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:18:49.789844 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-0ec9ce5f-2938-4ed0-a7d7-a373689b759f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:18:49.804233 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 11/24] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:18:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 12 17:18:49.847922 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e437c8c3-3442-474c-838a-a447071f6aa0 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:18:49.903761 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 14/25] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:18:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 12 17:18:49.936765 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec9ce5f-2938-4ed0-a7d7-a373689b759f None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:18:49.963095 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:18:50.062889 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e437c8c3-3442-474c-838a-a447071f6aa0 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:18:50.096643 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:18:50.126631 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec9ce5f-2938-4ed0-a7d7-a373689b759f None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:18:50.130067 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db350648-7277-4da5-9c01-5b30cc1e3746 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:18:50.130912 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db350648-7277-4da5-9c01-5b30cc1e3746 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:50.131200 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db350648-7277-4da5-9c01-5b30cc1e3746 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:50.131478 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db350648-7277-4da5-9c01-5b30cc1e3746 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:50.159279 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db350648-7277-4da5-9c01-5b30cc1e3746 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:18:50.160760 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db350648-7277-4da5-9c01-5b30cc1e3746 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:18:50.168062 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1813bb9d-3080-45ca-8596-d62952f35e5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:18:50.169012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1813bb9d-3080-45ca-8596-d62952f35e5a None admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:50.169498 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1813bb9d-3080-45ca-8596-d62952f35e5a None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:50.169977 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1813bb9d-3080-45ca-8596-d62952f35e5a None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:50.170973 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1813bb9d-3080-45ca-8596-d62952f35e5a None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:18:50.172591 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1813bb9d-3080-45ca-8596-d62952f35e5a None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:18:50.196362 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-db350648-7277-4da5-9c01-5b30cc1e3746 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:18:50.196362 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:18:50.196362 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:18:50.196362 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:18:50.196362 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:18:50.196362 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:18:50.196362 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:18:50.196362 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:18:50.196362 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:18:50.196362 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:18:50.196362 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:18:50.196362 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:18:50.196362 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:18:50.196362 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:18:50.196362 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:18:50.211624 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:18:50.211624 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:18:50.211624 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:18:50.211624 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:18:50.211624 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:18:50.211624 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:18:50.211624 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:18:50.211624 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:18:50.211624 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:18:50.211624 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:18:50.211624 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:18:50.211624 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:18:50.211624 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:18:50.211624 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:18:50.211624 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:18:50.211624 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:18:50.211624 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:18:50.212509 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:18:50.212509 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:18:50.212509 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:18:50.212509 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:18:50.212509 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 12/26] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:18:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 345 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:18:50.212509 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6daf35-1913-40dd-bdaa-72447476a26d None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:50.212509 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6daf35-1913-40dd-bdaa-72447476a26d None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:50.212509 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6daf35-1913-40dd-bdaa-72447476a26d None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:50.212509 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 13/27] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:18:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:18:50.212959 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6daf35-1913-40dd-bdaa-72447476a26d None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:18:50.372383 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:18:50.389712 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6daf35-1913-40dd-bdaa-72447476a26d None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:18:50.505168 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714f80d6-878c-4d8a-8e72-4ed0fd3876d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:18:50.506140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714f80d6-878c-4d8a-8e72-4ed0fd3876d8 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:50.506626 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714f80d6-878c-4d8a-8e72-4ed0fd3876d8 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:50.507857 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714f80d6-878c-4d8a-8e72-4ed0fd3876d8 None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:50.508831 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714f80d6-878c-4d8a-8e72-4ed0fd3876d8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:18:50.509322 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714f80d6-878c-4d8a-8e72-4ed0fd3876d8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:18:50.511000 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714f80d6-878c-4d8a-8e72-4ed0fd3876d8 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:18:50.526278 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 15/28] 10.222.0.16 () {60 vars in 1131 bytes} [Tue Jul 12 17:18:49 2022] POST /identity/v3/roles => generated 219 bytes in 590 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:18:50.530286 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5184420-bdfc-47c2-a172-79037bbc0dc9 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:50.530872 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5184420-bdfc-47c2-a172-79037bbc0dc9 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:50.532354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5184420-bdfc-47c2-a172-79037bbc0dc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:50.544289 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 14/29] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:18:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 342 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:18:50.552530 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714f80d6-878c-4d8a-8e72-4ed0fd3876d8 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:18:50.593572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-d5184420-bdfc-47c2-a172-79037bbc0dc9 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:18:50.596189 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-d5184420-bdfc-47c2-a172-79037bbc0dc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:18:50.814362 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 16/30] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:18:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 287 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:18:50.818105 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec2cf850-154f-419b-a128-d5fb71f9d25a None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:50.818105 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec2cf850-154f-419b-a128-d5fb71f9d25a None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:50.818105 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec2cf850-154f-419b-a128-d5fb71f9d25a None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:50.818105 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 17/31] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:18:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:18:50.825495 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef523ccf-1e2d-4dd6-b7e0-f40995462a8d None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:50.825833 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef523ccf-1e2d-4dd6-b7e0-f40995462a8d None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:50.826154 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef523ccf-1e2d-4dd6-b7e0-f40995462a8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:50.865892 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-ef523ccf-1e2d-4dd6-b7e0-f40995462a8d None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:18:50.867771 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714f80d6-878c-4d8a-8e72-4ed0fd3876d8 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:18:50.870246 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-ef523ccf-1e2d-4dd6-b7e0-f40995462a8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:18:50.913049 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2264264-fd95-4a5a-9e54-8d2544669973 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:18:50.914342 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2264264-fd95-4a5a-9e54-8d2544669973 None admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:50.914825 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2264264-fd95-4a5a-9e54-8d2544669973 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:50.915320 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2264264-fd95-4a5a-9e54-8d2544669973 None admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:50.916765 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2264264-fd95-4a5a-9e54-8d2544669973 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:18:50.918309 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2264264-fd95-4a5a-9e54-8d2544669973 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:18:50.972795 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:18:50.983043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 18/32] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:18:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 12 17:18:50.984199 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef523ccf-1e2d-4dd6-b7e0-f40995462a8d None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:18:51.115227 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef523ccf-1e2d-4dd6-b7e0-f40995462a8d None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:18:51.152016 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351945b2-6902-489e-81b9-ec071d4b0a90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:18:51.152814 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351945b2-6902-489e-81b9-ec071d4b0a90 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:51.153109 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351945b2-6902-489e-81b9-ec071d4b0a90 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:51.153391 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351945b2-6902-489e-81b9-ec071d4b0a90 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:51.164414 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351945b2-6902-489e-81b9-ec071d4b0a90 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:18:51.172287 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-351945b2-6902-489e-81b9-ec071d4b0a90 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:18:51.172287 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:18:51.172287 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:18:51.172287 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:18:51.172287 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:18:51.172287 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:18:51.172287 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:18:51.172287 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:18:51.172287 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:18:51.172287 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:18:51.172287 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:18:51.172287 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:18:51.172287 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:18:51.172287 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:18:51.172287 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:18:51.172287 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:18:51.173163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:18:51.173163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:18:51.173163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:18:51.173163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:18:51.173163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:18:51.173163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:18:51.173163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:18:51.173163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:18:51.173163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:18:51.173163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:18:51.173163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:18:51.173163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:18:51.173163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:18:51.173163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:18:51.173163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:18:51.173163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:18:51.173970 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:18:51.173970 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:18:51.173970 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:18:51.173970 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:18:51.173970 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:18:51.173970 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:18:51.173970 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:18:51.173970 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:18:51.173970 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:18:51.173970 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:18:51.173970 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:18:51.173970 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:18:51.173970 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 19/33] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Jul 12 17:18:50 2022] GET /identity/v3/roles/admin => generated 83 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:18:51.175191 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0517b55-96de-495e-a334-4be1a1f2b882 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:51.175191 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0517b55-96de-495e-a334-4be1a1f2b882 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:51.175417 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0517b55-96de-495e-a334-4be1a1f2b882 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:51.176871 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 20/34] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:18:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:18:51.177339 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0517b55-96de-495e-a334-4be1a1f2b882 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:18:51.181191 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 15/35] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Jul 12 17:18:50 2022] POST /identity/v3/projects => generated 301 bytes in 627 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:18:51.181191 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2c3ebf-6cbb-4742-9ca3-8ce1cb1ee60b None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:51.181191 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2c3ebf-6cbb-4742-9ca3-8ce1cb1ee60b None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:51.181514 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2c3ebf-6cbb-4742-9ca3-8ce1cb1ee60b None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:51.183043 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 16/36] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:18:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:18:51.189180 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab3898f-0cb0-44e5-8603-79e3403b5390 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:51.189482 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab3898f-0cb0-44e5-8603-79e3403b5390 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:51.189771 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dab3898f-0cb0-44e5-8603-79e3403b5390 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:51.208965 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-dab3898f-0cb0-44e5-8603-79e3403b5390 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:18:51.211348 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-dab3898f-0cb0-44e5-8603-79e3403b5390 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:18:51.215685 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:18:51.254398 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:18:51.294026 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 17/37] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:18:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 12 17:18:51.294783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab3898f-0cb0-44e5-8603-79e3403b5390 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:18:51.318246 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0517b55-96de-495e-a334-4be1a1f2b882 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:18:51.363937 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b8e90b-b511-4b9b-aebe-35ef5bf0ec2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:18:51.364620 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b8e90b-b511-4b9b-aebe-35ef5bf0ec2a None admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:51.364917 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b8e90b-b511-4b9b-aebe-35ef5bf0ec2a None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:51.365205 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b8e90b-b511-4b9b-aebe-35ef5bf0ec2a None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:51.366019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5b8e90b-b511-4b9b-aebe-35ef5bf0ec2a None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:18:51.367806 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5b8e90b-b511-4b9b-aebe-35ef5bf0ec2a None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:18:51.410284 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dab3898f-0cb0-44e5-8603-79e3403b5390 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:18:51.443622 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1fb099-da54-4b06-915c-9dcecdd7f2b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:18:51.444846 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1fb099-da54-4b06-915c-9dcecdd7f2b4 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:51.445140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1fb099-da54-4b06-915c-9dcecdd7f2b4 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:51.445426 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1fb099-da54-4b06-915c-9dcecdd7f2b4 None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:51.445936 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 21/38] 10.222.0.16 () {60 vars in 1131 bytes} [Tue Jul 12 17:18:51 2022] POST /identity/v3/roles => generated 225 bytes in 269 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:18:51.446196 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f1fb099-da54-4b06-915c-9dcecdd7f2b4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:18:51.446545 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f1fb099-da54-4b06-915c-9dcecdd7f2b4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:18:51.447466 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e94beb-9ec6-4637-a237-c3c87ad896fb None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:51.447806 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e94beb-9ec6-4637-a237-c3c87ad896fb None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:51.448077 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f1fb099-da54-4b06-915c-9dcecdd7f2b4 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:18:51.448170 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e94beb-9ec6-4637-a237-c3c87ad896fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:51.468018 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 18/39] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:18:51 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:18:51.468018 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1fb099-da54-4b06-915c-9dcecdd7f2b4 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:18:51.468018 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:18:51.475062 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-f6e94beb-9ec6-4637-a237-c3c87ad896fb None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:18:51.476546 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-f6e94beb-9ec6-4637-a237-c3c87ad896fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:18:51.566521 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 22/40] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:18:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:18:51.566521 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e94beb-9ec6-4637-a237-c3c87ad896fb None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:18:51.600173 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1fb099-da54-4b06-915c-9dcecdd7f2b4 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:18:51.620041 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:18:51.642206 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652b9b3a-7256-4834-81d3-9b15c8da353e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:18:51.643134 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652b9b3a-7256-4834-81d3-9b15c8da353e None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:51.643685 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652b9b3a-7256-4834-81d3-9b15c8da353e None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:51.643788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652b9b3a-7256-4834-81d3-9b15c8da353e None admin] PATH_INFO: `/v3/domains/default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:51.645678 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652b9b3a-7256-4834-81d3-9b15c8da353e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:18:51.648539 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652b9b3a-7256-4834-81d3-9b15c8da353e None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:18:51.650373 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 19/41] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:18:51 2022] GET /identity/v3/domains/default => generated 199 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:18:51.651735 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652b9b3a-7256-4834-81d3-9b15c8da353e None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:18:51.687250 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e94beb-9ec6-4637-a237-c3c87ad896fb None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:18:51.723342 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d615a1f-af87-4ae6-85a9-9c940133883e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:18:51.724225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d615a1f-af87-4ae6-85a9-9c940133883e None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:51.724519 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d615a1f-af87-4ae6-85a9-9c940133883e None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:51.724809 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d615a1f-af87-4ae6-85a9-9c940133883e None admin] PATH_INFO: `/v3/users/df85f597d0ba471381a5a8f9b6886705` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:51.725092 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:18:51.726716 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d615a1f-af87-4ae6-85a9-9c940133883e None admin] RBAC: Authorizing `identity:get_user(user_id=df85f597d0ba471381a5a8f9b6886705)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:18:51.728230 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d615a1f-af87-4ae6-85a9-9c940133883e None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:18:51.729904 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 23/42] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:18:51 2022] GET /identity/v3/users/df85f597d0ba471381a5a8f9b6886705 => generated 247 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:18:51.730681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d615a1f-af87-4ae6-85a9-9c940133883e None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:18:51.757504 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652b9b3a-7256-4834-81d3-9b15c8da353e None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:18:51.790759 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c124b4a-69b2-4838-aa9b-45be08c9e06c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:18:51.791435 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c124b4a-69b2-4838-aa9b-45be08c9e06c None admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:51.791729 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c124b4a-69b2-4838-aa9b-45be08c9e06c None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:51.792015 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c124b4a-69b2-4838-aa9b-45be08c9e06c None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:51.792781 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c124b4a-69b2-4838-aa9b-45be08c9e06c None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:18:51.794277 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c124b4a-69b2-4838-aa9b-45be08c9e06c None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:18:51.831727 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d615a1f-af87-4ae6-85a9-9c940133883e None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:18:51.845001 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:18:51.867418 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbbb7da1-e591-4f83-9ec4-84b6179d2408 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:18:51.868102 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbb7da1-e591-4f83-9ec4-84b6179d2408 None admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:51.868393 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbb7da1-e591-4f83-9ec4-84b6179d2408 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:51.868676 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbb7da1-e591-4f83-9ec4-84b6179d2408 None admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:51.869388 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbbb7da1-e591-4f83-9ec4-84b6179d2408 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:18:51.870929 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbbb7da1-e591-4f83-9ec4-84b6179d2408 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:18:51.885981 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 20/43] 10.222.0.16 () {60 vars in 1131 bytes} [Tue Jul 12 17:18:51 2022] POST /identity/v3/roles => generated 223 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:18:51.886970 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c124b4a-69b2-4838-aa9b-45be08c9e06c None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:18:51.981566 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 24/44] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Jul 12 17:18:51 2022] POST /identity/v3/projects => generated 312 bytes in 251 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:18:52.012047 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c124b4a-69b2-4838-aa9b-45be08c9e06c None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:18:52.049630 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:18:52.057290 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e623b297-cf23-4d48-9aec-83752ac9f467 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:18:52.058659 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e623b297-cf23-4d48-9aec-83752ac9f467 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:52.059150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e623b297-cf23-4d48-9aec-83752ac9f467 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:52.059437 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e623b297-cf23-4d48-9aec-83752ac9f467 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:52.060768 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e623b297-cf23-4d48-9aec-83752ac9f467 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:18:52.062282 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e623b297-cf23-4d48-9aec-83752ac9f467 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:18:52.064212 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 21/45] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:18:51 2022] GET /identity/v3/domains/default => generated 199 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:18:52.070691 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbbb7da1-e591-4f83-9ec4-84b6179d2408 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:18:52.282245 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbbb7da1-e591-4f83-9ec4-84b6179d2408 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:18:52.316309 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2bf686-f619-48e7-afd5-c6024d7372b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:18:52.317182 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2bf686-f619-48e7-afd5-c6024d7372b6 None admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:52.317467 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2bf686-f619-48e7-afd5-c6024d7372b6 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:52.317746 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2bf686-f619-48e7-afd5-c6024d7372b6 None admin] PATH_INFO: `/v3/domains/default/users/df85f597d0ba471381a5a8f9b6886705/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:52.320252 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da2bf686-f619-48e7-afd5-c6024d7372b6 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=df85f597d0ba471381a5a8f9b6886705, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:18:52.321707 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da2bf686-f619-48e7-afd5-c6024d7372b6 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:18:52.425265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 25/46] 10.222.0.16 () {58 vars in 1361 bytes} [Tue Jul 12 17:18:52 2022] PUT /identity/v3/domains/default/users/df85f597d0ba471381a5a8f9b6886705/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 355 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:18:54.060389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98e0419-f09d-443c-939d-b29cef0d441e None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:54.060934 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98e0419-f09d-443c-939d-b29cef0d441e None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:54.060934 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98e0419-f09d-443c-939d-b29cef0d441e None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:54.062329 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 22/47] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:18:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 12 17:18:54.067678 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822cb27d-3642-4fd5-90ed-c82900b1425b None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:54.068097 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822cb27d-3642-4fd5-90ed-c82900b1425b None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:54.068400 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822cb27d-3642-4fd5-90ed-c82900b1425b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:54.086330 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-822cb27d-3642-4fd5-90ed-c82900b1425b None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:18:54.087566 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-822cb27d-3642-4fd5-90ed-c82900b1425b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:18:54.105081 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:18:54.158918 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 26/48] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:18:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:18:54.182396 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e98e0419-f09d-443c-939d-b29cef0d441e None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:18:54.286420 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e98e0419-f09d-443c-939d-b29cef0d441e None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:18:54.322275 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0eeaad-534b-455f-9eef-b44f29a3f47f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:18:54.323039 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0eeaad-534b-455f-9eef-b44f29a3f47f None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:54.323330 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0eeaad-534b-455f-9eef-b44f29a3f47f None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:54.323643 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0eeaad-534b-455f-9eef-b44f29a3f47f None admin] PATH_INFO: `/v3/domains/default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:54.324868 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0eeaad-534b-455f-9eef-b44f29a3f47f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:18:54.326330 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0eeaad-534b-455f-9eef-b44f29a3f47f None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:18:54.328188 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 23/49] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:18:54 2022] GET /identity/v3/domains/default => generated 199 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:18:54.333580 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822cb27d-3642-4fd5-90ed-c82900b1425b None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:18:54.433234 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822cb27d-3642-4fd5-90ed-c82900b1425b None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:18:54.437803 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6bb3e1-d978-4a47-bc1a-25723706b2ba None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:54.438098 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6bb3e1-d978-4a47-bc1a-25723706b2ba None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:54.438386 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6bb3e1-d978-4a47-bc1a-25723706b2ba None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:54.439805 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 24/50] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:18:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 12 17:18:54.444877 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0382a2a4-d03d-44da-aeab-952a24719693 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:54.445178 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0382a2a4-d03d-44da-aeab-952a24719693 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:54.445465 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0382a2a4-d03d-44da-aeab-952a24719693 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:54.462274 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-0382a2a4-d03d-44da-aeab-952a24719693 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:18:54.463521 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-0382a2a4-d03d-44da-aeab-952a24719693 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:18:54.467558 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae7c978-5303-48c1-84a5-a89221ecd457 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:18:54.468224 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae7c978-5303-48c1-84a5-a89221ecd457 None admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:54.468509 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae7c978-5303-48c1-84a5-a89221ecd457 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:54.468783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae7c978-5303-48c1-84a5-a89221ecd457 None admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:54.469481 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae7c978-5303-48c1-84a5-a89221ecd457 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:18:54.471092 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae7c978-5303-48c1-84a5-a89221ecd457 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:18:54.524149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:18:54.531361 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 25/51] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:18:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 12 17:18:54.555732 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0382a2a4-d03d-44da-aeab-952a24719693 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:18:54.567513 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 27/52] 10.222.0.16 () {60 vars in 1140 bytes} [Tue Jul 12 17:18:54 2022] POST /identity/v3/projects => generated 298 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:18:54.670315 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0382a2a4-d03d-44da-aeab-952a24719693 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:18:54.707017 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c702e5-4a39-4395-b033-4c230a209358 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:18:54.707831 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c702e5-4a39-4395-b033-4c230a209358 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:54.708120 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c702e5-4a39-4395-b033-4c230a209358 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:54.708400 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c702e5-4a39-4395-b033-4c230a209358 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:54.720915 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c702e5-4a39-4395-b033-4c230a209358 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:18:54.730562 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-58c702e5-4a39-4395-b033-4c230a209358 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:18:54.730562 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:18:54.730562 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:18:54.730562 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:18:54.730562 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:18:54.730562 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:18:54.730562 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:18:54.730562 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:18:54.730562 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:18:54.730562 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:18:54.730562 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:18:54.730562 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:18:54.730562 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:18:54.730562 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:18:54.730562 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:18:54.730562 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:18:54.731487 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:18:54.731487 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:18:54.731487 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:18:54.731487 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:18:54.731487 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:18:54.731487 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:18:54.731487 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:18:54.731487 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:18:54.731487 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:18:54.731487 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:18:54.731487 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:18:54.731487 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:18:54.731487 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:18:54.731487 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:18:54.731487 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:18:54.731487 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:18:54.732295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:18:54.732295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:18:54.732295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:18:54.732295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:18:54.732295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:18:54.732295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:18:54.732295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:18:54.732295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:18:54.732295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:18:54.732295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:18:54.732295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:18:54.732295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:18:54.732295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 26/53] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Jul 12 17:18:54 2022] GET /identity/v3/roles/admin => generated 83 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:18:54.739406 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae7c978-5303-48c1-84a5-a89221ecd457 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:18:54.751197 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:18:54.840706 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae7c978-5303-48c1-84a5-a89221ecd457 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:18:54.875257 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb763934-7497-4880-b014-00696302443f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:18:54.875943 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb763934-7497-4880-b014-00696302443f None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:54.876239 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb763934-7497-4880-b014-00696302443f None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:54.876525 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb763934-7497-4880-b014-00696302443f None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:54.877287 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb763934-7497-4880-b014-00696302443f None admin] RBAC: Adding query filter params (name=admin) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:18:54.877656 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb763934-7497-4880-b014-00696302443f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:18:54.879058 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb763934-7497-4880-b014-00696302443f None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:18:54.888138 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 28/54] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:18:54 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:18:54.893657 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c702e5-4a39-4395-b033-4c230a209358 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:18:54.961760 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:18:55.112873 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c702e5-4a39-4395-b033-4c230a209358 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:18:55.148670 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698f3198-1e88-4c74-ab77-f3591550bd3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:18:55.149554 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698f3198-1e88-4c74-ab77-f3591550bd3b None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:55.149856 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698f3198-1e88-4c74-ab77-f3591550bd3b None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:55.150137 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698f3198-1e88-4c74-ab77-f3591550bd3b None admin] PATH_INFO: `/v3/users/df85f597d0ba471381a5a8f9b6886705` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:55.151643 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-698f3198-1e88-4c74-ab77-f3591550bd3b None admin] RBAC: Authorizing `identity:get_user(user_id=df85f597d0ba471381a5a8f9b6886705)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:18:55.153096 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-698f3198-1e88-4c74-ab77-f3591550bd3b None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:18:55.154772 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 27/55] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:18:54 2022] GET /identity/v3/users/df85f597d0ba471381a5a8f9b6886705 => generated 247 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:18:55.160101 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb763934-7497-4880-b014-00696302443f None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:18:55.226600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:18:55.260426 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb763934-7497-4880-b014-00696302443f None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:18:55.294186 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6b2bae-95b0-48b5-b2bd-64174edb08c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:18:55.295065 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6b2bae-95b0-48b5-b2bd-64174edb08c8 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:55.295329 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6b2bae-95b0-48b5-b2bd-64174edb08c8 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:55.295609 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6b2bae-95b0-48b5-b2bd-64174edb08c8 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:55.296795 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f6b2bae-95b0-48b5-b2bd-64174edb08c8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:18:55.298254 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f6b2bae-95b0-48b5-b2bd-64174edb08c8 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:18:55.299972 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 29/56] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:18:55 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:18:55.305662 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698f3198-1e88-4c74-ab77-f3591550bd3b None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:18:55.407459 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698f3198-1e88-4c74-ab77-f3591550bd3b None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:18:55.430989 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:18:55.441409 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc9a52db-78e6-4560-aab8-57d013453af5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:18:55.442117 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9a52db-78e6-4560-aab8-57d013453af5 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:55.442395 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9a52db-78e6-4560-aab8-57d013453af5 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:55.442704 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc9a52db-78e6-4560-aab8-57d013453af5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:55.443718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc9a52db-78e6-4560-aab8-57d013453af5 None admin] RBAC: Adding query filter params (role.id=eb05bf286c894ea9bf44a6c023a4323b, scope.domain.id=default, user.id=df85f597d0ba471381a5a8f9b6886705) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:18:55.444567 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc9a52db-78e6-4560-aab8-57d013453af5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:18:55.446168 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc9a52db-78e6-4560-aab8-57d013453af5 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:18:55.454847 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 28/57] 10.222.0.16 () {56 vars in 1322 bytes} [Tue Jul 12 17:18:55 2022] GET /identity/v3/role_assignments?user.id=df85f597d0ba471381a5a8f9b6886705&scope.domain.id=default&role.id=eb05bf286c894ea9bf44a6c023a4323b => generated 530 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:18:56.635484 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8de5cd-d5dc-448e-8ff5-56154431bbc9 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:56.635985 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8de5cd-d5dc-448e-8ff5-56154431bbc9 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:56.636071 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8de5cd-d5dc-448e-8ff5-56154431bbc9 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:56.637419 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 30/58] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:18:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:18:56.642531 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c722b45-0ff9-4003-870f-8d0506e85ca9 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:56.642988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c722b45-0ff9-4003-870f-8d0506e85ca9 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:56.643124 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c722b45-0ff9-4003-870f-8d0506e85ca9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:56.659736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-2c722b45-0ff9-4003-870f-8d0506e85ca9 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:18:56.660955 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-2c722b45-0ff9-4003-870f-8d0506e85ca9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:18:56.727773 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 29/59] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:18:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:18:56.752413 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8de5cd-d5dc-448e-8ff5-56154431bbc9 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:18:56.851400 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8de5cd-d5dc-448e-8ff5-56154431bbc9 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:18:56.884379 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0ff4f5-801f-4a39-ad75-bf275493919c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:18:56.885153 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0ff4f5-801f-4a39-ad75-bf275493919c None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:56.885440 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0ff4f5-801f-4a39-ad75-bf275493919c None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:56.885719 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0ff4f5-801f-4a39-ad75-bf275493919c None admin] PATH_INFO: `/v3/domains/default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:56.886907 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df0ff4f5-801f-4a39-ad75-bf275493919c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:18:56.888430 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df0ff4f5-801f-4a39-ad75-bf275493919c None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:18:56.890070 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 31/60] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:18:56 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:18:56.895583 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c722b45-0ff9-4003-870f-8d0506e85ca9 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:18:56.997797 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c722b45-0ff9-4003-870f-8d0506e85ca9 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:18:57.015338 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:18:57.031897 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6190956f-5bdb-4ed2-b4bc-59bce7882eb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:18:57.032580 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6190956f-5bdb-4ed2-b4bc-59bce7882eb4 None admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:18:57.032874 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6190956f-5bdb-4ed2-b4bc-59bce7882eb4 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:18:57.033160 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6190956f-5bdb-4ed2-b4bc-59bce7882eb4 None admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:18:57.033889 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6190956f-5bdb-4ed2-b4bc-59bce7882eb4 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:18:57.035382 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6190956f-5bdb-4ed2-b4bc-59bce7882eb4 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:18:57.153061 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 30/61] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Jul 12 17:18:56 2022] POST /identity/v3/users => generated 275 bytes in 258 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:19:00.249448 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6dd77e-935e-40bc-b564-3da899c491bc None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:00.250036 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6dd77e-935e-40bc-b564-3da899c491bc None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:00.250036 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6dd77e-935e-40bc-b564-3da899c491bc None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:00.251469 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 32/62] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:00.257148 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc8d335-3eee-47b9-bd86-8543d993aff2 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:00.257639 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc8d335-3eee-47b9-bd86-8543d993aff2 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:00.257756 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc8d335-3eee-47b9-bd86-8543d993aff2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:00.284186 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-dbc8d335-3eee-47b9-bd86-8543d993aff2 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:00.285409 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-dbc8d335-3eee-47b9-bd86-8543d993aff2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:00.346618 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b20917e-d604-4053-94ca-deae84bdb4d0 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:00.347265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b20917e-d604-4053-94ca-deae84bdb4d0 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:00.347762 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b20917e-d604-4053-94ca-deae84bdb4d0 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:00.349386 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 33/63] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:00.355280 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7c49c1-1e35-48e9-abdd-e8377e9a1f27 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:00.355878 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7c49c1-1e35-48e9-abdd-e8377e9a1f27 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:00.356387 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7c49c1-1e35-48e9-abdd-e8377e9a1f27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:00.364465 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:00.376277 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-fe7c49c1-1e35-48e9-abdd-e8377e9a1f27 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:00.377775 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-fe7c49c1-1e35-48e9-abdd-e8377e9a1f27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:00.400764 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 31/64] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:00.474399 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc8d335-3eee-47b9-bd86-8543d993aff2 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:00.475954 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 34/65] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:00.498214 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7c49c1-1e35-48e9-abdd-e8377e9a1f27 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:00.606566 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7c49c1-1e35-48e9-abdd-e8377e9a1f27 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:00.649916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbd11d4-cfee-4914-90e4-ad0db061fa33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:00.650764 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbd11d4-cfee-4914-90e4-ad0db061fa33 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:00.651060 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbd11d4-cfee-4914-90e4-ad0db061fa33 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:00.651343 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbd11d4-cfee-4914-90e4-ad0db061fa33 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:00.662789 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:00.669496 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbbd11d4-cfee-4914-90e4-ad0db061fa33 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:00.677671 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-bbbd11d4-cfee-4914-90e4-ad0db061fa33 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 12 17:19:00.677671 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:00.677671 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:00.677671 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:00.677671 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:00.677671 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:00.677671 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:00.677671 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:00.677671 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:00.677671 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:00.677671 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:00.677671 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:00.677671 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:00.677671 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:00.677671 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:00.677671 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:00.679490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:00.679490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:00.679490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:00.679490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:00.679490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:00.679490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:00.679490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:00.679490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:00.679490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:00.679490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:00.679490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:00.679490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:00.679490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:00.679490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:00.679490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:00.679490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:00.680501 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:00.680501 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:00.680501 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:00.680501 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:00.680501 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:00.680501 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:00.680501 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:00.680501 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:00.680501 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:00.680501 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:00.680501 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 12 17:19:00.680501 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:19:00.682510 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 35/66] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:00 2022] GET /identity/v3/roles/member => generated 84 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:19:00.688748 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbd11d4-cfee-4914-90e4-ad0db061fa33 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:00.707424 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc8d335-3eee-47b9-bd86-8543d993aff2 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:00.715636 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:00.800500 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbd11d4-cfee-4914-90e4-ad0db061fa33 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:00.804061 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbceeef3-0223-448a-9e3c-7d4db2d6c798 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:00.805138 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbceeef3-0223-448a-9e3c-7d4db2d6c798 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:00.805631 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbceeef3-0223-448a-9e3c-7d4db2d6c798 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:00.806107 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbceeef3-0223-448a-9e3c-7d4db2d6c798 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:00.817220 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbceeef3-0223-448a-9e3c-7d4db2d6c798 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:00.826036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-fbceeef3-0223-448a-9e3c-7d4db2d6c798 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:19:00.826036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:00.826036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:00.826036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:00.826036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:00.826036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:00.826036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:00.826036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:00.826036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:00.826036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:00.826036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:00.826036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:00.826036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:00.826036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:00.826036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:00.826036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:00.826932 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:00.826932 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:00.826932 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:00.826932 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:00.826932 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:00.826932 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:00.826932 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:00.826932 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:00.826932 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:00.826932 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:00.826932 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:00.826932 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:00.826932 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:00.826932 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:00.826932 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:00.826932 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:00.827740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:00.827740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:00.827740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:00.827740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:00.827740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:00.827740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:00.827740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:00.827740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:00.827740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:00.827740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:00.827740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:19:00.827740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:00.830374 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 32/67] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Jul 12 17:19:00 2022] GET /identity/v3/roles/admin => generated 83 bytes in 356 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:19:00.831846 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb06397-6518-4168-b05a-6056bf336f48 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:00.832347 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb06397-6518-4168-b05a-6056bf336f48 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:00.832835 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb06397-6518-4168-b05a-6056bf336f48 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:00.835004 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 33/68] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:00.839647 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb06397-6518-4168-b05a-6056bf336f48 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:00.842882 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8320fc-45bb-4591-80bf-a6d0159d62ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:00.844672 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8320fc-45bb-4591-80bf-a6d0159d62ac None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:00.845183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8320fc-45bb-4591-80bf-a6d0159d62ac None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:00.845688 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8320fc-45bb-4591-80bf-a6d0159d62ac None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:00.846681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db8320fc-45bb-4591-80bf-a6d0159d62ac None admin] RBAC: Adding query filter params (name=member) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:00.847193 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db8320fc-45bb-4591-80bf-a6d0159d62ac None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:00.848797 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db8320fc-45bb-4591-80bf-a6d0159d62ac None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:00.859154 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 36/69] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Jul 12 17:19:00 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:19:00.860596 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cd064b-e131-48f5-ab8d-194c4e5178b3 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:00.860891 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cd064b-e131-48f5-ab8d-194c4e5178b3 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:00.861173 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cd064b-e131-48f5-ab8d-194c4e5178b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:00.878394 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-57cd064b-e131-48f5-ab8d-194c4e5178b3 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:00.879708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-57cd064b-e131-48f5-ab8d-194c4e5178b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:00.936709 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:00.953185 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 37/70] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:00.954450 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b0f87d-8db9-4d19-98a4-7435240c1e6d None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:00.954761 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b0f87d-8db9-4d19-98a4-7435240c1e6d None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:00.955083 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b0f87d-8db9-4d19-98a4-7435240c1e6d None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:00.956364 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 38/71] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:00.957063 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b0f87d-8db9-4d19-98a4-7435240c1e6d None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:00.972172 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb06397-6518-4168-b05a-6056bf336f48 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:01.014933 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22147ece-53a9-4232-a073-d155997ffa90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:01.015610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22147ece-53a9-4232-a073-d155997ffa90 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:01.015899 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22147ece-53a9-4232-a073-d155997ffa90 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:01.016180 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22147ece-53a9-4232-a073-d155997ffa90 None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:01.017029 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22147ece-53a9-4232-a073-d155997ffa90 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:01.017350 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22147ece-53a9-4232-a073-d155997ffa90 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:01.018782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22147ece-53a9-4232-a073-d155997ffa90 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:01.028630 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 34/72] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:00 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:01.029985 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808086ae-9c3c-460e-861b-ce911e1d8530 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:01.030286 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808086ae-9c3c-460e-861b-ce911e1d8530 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:01.030595 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808086ae-9c3c-460e-861b-ce911e1d8530 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:01.048216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-808086ae-9c3c-460e-861b-ce911e1d8530 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:01.049493 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-808086ae-9c3c-460e-861b-ce911e1d8530 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:01.071899 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:01.077617 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b0f87d-8db9-4d19-98a4-7435240c1e6d None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:01.112327 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f6ec3e-c786-4589-bb1e-94bf62d4b12a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:01.113139 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f6ec3e-c786-4589-bb1e-94bf62d4b12a None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:01.113434 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f6ec3e-c786-4589-bb1e-94bf62d4b12a None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:01.113722 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f6ec3e-c786-4589-bb1e-94bf62d4b12a None admin] PATH_INFO: `/v3/users/dbc90e3cb55f431e91cd23da7b8958f2` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:01.117997 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 35/73] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:01.119190 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808086ae-9c3c-460e-861b-ce911e1d8530 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:01.139215 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64f6ec3e-c786-4589-bb1e-94bf62d4b12a None admin] RBAC: Authorizing `identity:get_user(user_id=dbc90e3cb55f431e91cd23da7b8958f2)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:01.140811 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64f6ec3e-c786-4589-bb1e-94bf62d4b12a None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:01.142584 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 39/74] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:19:00 2022] GET /identity/v3/users/dbc90e3cb55f431e91cd23da7b8958f2 => generated 275 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:01.143355 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f6ec3e-c786-4589-bb1e-94bf62d4b12a None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:01.189205 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:01.225984 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808086ae-9c3c-460e-861b-ce911e1d8530 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:01.246857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f6ec3e-c786-4589-bb1e-94bf62d4b12a None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:01.260760 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f35006-56f9-4004-a135-aa1fb212b2fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:01.261567 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f35006-56f9-4004-a135-aa1fb212b2fc None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:01.261854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f35006-56f9-4004-a135-aa1fb212b2fc None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:01.262137 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f35006-56f9-4004-a135-aa1fb212b2fc None admin] PATH_INFO: `/v3/domains/default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:01.263335 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f35006-56f9-4004-a135-aa1fb212b2fc None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:01.264783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f35006-56f9-4004-a135-aa1fb212b2fc None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:01.266455 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 36/75] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:19:01 2022] GET /identity/v3/domains/default => generated 199 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:01.271246 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29cfba10-c7cf-4928-a1d4-82cdab7d19db None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:01.271246 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29cfba10-c7cf-4928-a1d4-82cdab7d19db None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:01.271246 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29cfba10-c7cf-4928-a1d4-82cdab7d19db None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:01.271246 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 37/76] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:01 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:01.271246 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29cfba10-c7cf-4928-a1d4-82cdab7d19db None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:01.285875 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92536db-c48c-434b-89e3-d5f8d1ca7c50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:01.286798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92536db-c48c-434b-89e3-d5f8d1ca7c50 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:01.287134 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92536db-c48c-434b-89e3-d5f8d1ca7c50 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:01.287416 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92536db-c48c-434b-89e3-d5f8d1ca7c50 None admin] PATH_INFO: `/v3/users/df85f597d0ba471381a5a8f9b6886705` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:01.288643 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92536db-c48c-434b-89e3-d5f8d1ca7c50 None admin] RBAC: Authorizing `identity:get_user(user_id=df85f597d0ba471381a5a8f9b6886705)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:01.290104 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92536db-c48c-434b-89e3-d5f8d1ca7c50 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:01.291778 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 40/77] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:19:01 2022] GET /identity/v3/users/df85f597d0ba471381a5a8f9b6886705 => generated 247 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:01.292538 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92536db-c48c-434b-89e3-d5f8d1ca7c50 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:01.300739 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:01.378836 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29cfba10-c7cf-4928-a1d4-82cdab7d19db None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:01.395313 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92536db-c48c-434b-89e3-d5f8d1ca7c50 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:01.409146 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:01.414128 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f34f78-ebf8-443e-887c-f69ce52ce692 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:01.415013 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f34f78-ebf8-443e-887c-f69ce52ce692 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:01.415309 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f34f78-ebf8-443e-887c-f69ce52ce692 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:01.415593 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f34f78-ebf8-443e-887c-f69ce52ce692 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:01.426168 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f34f78-ebf8-443e-887c-f69ce52ce692 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:01.430882 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48b0815-a3ae-4b27-b86a-eed04a89920b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:01.431566 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48b0815-a3ae-4b27-b86a-eed04a89920b None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:01.431859 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48b0815-a3ae-4b27-b86a-eed04a89920b None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:01.432139 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48b0815-a3ae-4b27-b86a-eed04a89920b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:01.434822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-28f34f78-ebf8-443e-887c-f69ce52ce692 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 12 17:19:01.434822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:01.434822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:01.434822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:01.434822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:01.434822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:01.434822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:01.434822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:01.434822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:01.434822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:01.434822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:01.434822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:01.434822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:01.434822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:01.434822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:01.434822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:01.435696 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:01.435696 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:01.435696 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:01.435696 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:01.435696 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:01.435696 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:01.435696 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:01.435696 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:01.435696 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:01.435696 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:01.435696 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:01.435696 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:01.435696 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:01.435696 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:01.435696 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:01.435696 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:01.436460 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:01.436460 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:01.436460 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:01.436460 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:01.436460 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:01.436460 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:01.436460 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:01.436460 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:01.436460 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:01.436460 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:01.436460 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 12 17:19:01.436460 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:01.436460 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 38/78] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Jul 12 17:19:01 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:01.437097 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f34f78-ebf8-443e-887c-f69ce52ce692 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:01.501604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b48b0815-a3ae-4b27-b86a-eed04a89920b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:01.503115 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b48b0815-a3ae-4b27-b86a-eed04a89920b None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:01.533729 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:01.550268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f34f78-ebf8-443e-887c-f69ce52ce692 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:01.589851 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38daa56-ab48-488c-9899-bb404fc78266 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:01.590562 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38daa56-ab48-488c-9899-bb404fc78266 None admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:01.590858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38daa56-ab48-488c-9899-bb404fc78266 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:01.591146 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f38daa56-ab48-488c-9899-bb404fc78266 None admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:01.591872 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f38daa56-ab48-488c-9899-bb404fc78266 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:01.593346 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f38daa56-ab48-488c-9899-bb404fc78266 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:01.614569 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 41/79] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:19:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 12 17:19:01.616805 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a3fadd-dfff-48f8-9133-81246da42c25 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:01.616805 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a3fadd-dfff-48f8-9133-81246da42c25 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:01.617179 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a3fadd-dfff-48f8-9133-81246da42c25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:01.637672 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-e1a3fadd-dfff-48f8-9133-81246da42c25 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:01.638872 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-e1a3fadd-dfff-48f8-9133-81246da42c25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:01.646629 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:01.672999 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 39/80] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Jul 12 17:19:01 2022] POST /identity/v3/users => generated 289 bytes in 236 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:19:01.674097 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38daa56-ab48-488c-9899-bb404fc78266 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:01.718716 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 42/81] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:01.718716 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a3fadd-dfff-48f8-9133-81246da42c25 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:01.794008 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f38daa56-ab48-488c-9899-bb404fc78266 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:01.834849 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b04a7332-744f-4f3a-bfbb-6a5451613d04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:01.835498 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04a7332-744f-4f3a-bfbb-6a5451613d04 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:01.835791 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04a7332-744f-4f3a-bfbb-6a5451613d04 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:01.837059 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04a7332-744f-4f3a-bfbb-6a5451613d04 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:01.849207 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:01.853307 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a3fadd-dfff-48f8-9133-81246da42c25 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:01.900305 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f0dbe9-77c5-40b2-9804-9bdc8fd44b3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:01.900925 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f0dbe9-77c5-40b2-9804-9bdc8fd44b3c None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:01.901217 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f0dbe9-77c5-40b2-9804-9bdc8fd44b3c None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:01.901505 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f0dbe9-77c5-40b2-9804-9bdc8fd44b3c None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:01.902275 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3f0dbe9-77c5-40b2-9804-9bdc8fd44b3c None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:01.902631 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3f0dbe9-77c5-40b2-9804-9bdc8fd44b3c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:01.904037 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3f0dbe9-77c5-40b2-9804-9bdc8fd44b3c None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:01.914218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 43/82] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:19:01 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:01.915498 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f0dbe9-77c5-40b2-9804-9bdc8fd44b3c None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:01.919338 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:01.931893 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b04a7332-744f-4f3a-bfbb-6a5451613d04 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:01.934241 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b04a7332-744f-4f3a-bfbb-6a5451613d04 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:02.106680 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f0dbe9-77c5-40b2-9804-9bdc8fd44b3c None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:02.127173 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 40/83] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:19:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 453 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 12 17:19:02.128123 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b04a7332-744f-4f3a-bfbb-6a5451613d04 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:02.149478 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f0a4fd-266d-408c-a316-52964c0c9ef7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:02.150477 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f0a4fd-266d-408c-a316-52964c0c9ef7 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:02.150723 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f0a4fd-266d-408c-a316-52964c0c9ef7 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:02.151006 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f0a4fd-266d-408c-a316-52964c0c9ef7 None admin] PATH_INFO: `/v3/projects/4857259d2a364dff960547dc0aac75e0` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:02.152233 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f0a4fd-266d-408c-a316-52964c0c9ef7 None admin] RBAC: Authorizing `identity:get_project(project_id=4857259d2a364dff960547dc0aac75e0)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:02.153703 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f0a4fd-266d-408c-a316-52964c0c9ef7 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:02.155543 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 44/84] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:01 2022] GET /identity/v3/projects/4857259d2a364dff960547dc0aac75e0 => generated 298 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:02.156329 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f0a4fd-266d-408c-a316-52964c0c9ef7 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:02.191840 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:02.248563 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:02.250966 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b04a7332-744f-4f3a-bfbb-6a5451613d04 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:02.288812 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f0a4fd-266d-408c-a316-52964c0c9ef7 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:02.290914 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82765df-19ef-4ba5-9241-9466aba93b82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:02.291730 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82765df-19ef-4ba5-9241-9466aba93b82 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:02.292024 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82765df-19ef-4ba5-9241-9466aba93b82 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:02.292305 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82765df-19ef-4ba5-9241-9466aba93b82 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:02.308082 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e82765df-19ef-4ba5-9241-9466aba93b82 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:02.316336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-e82765df-19ef-4ba5-9241-9466aba93b82 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 12 17:19:02.316336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:02.316336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:02.316336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:02.316336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:02.316336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:02.316336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:02.316336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:02.316336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:02.316336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:02.316336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:02.316336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:02.316336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:02.316336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:02.316336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:02.316336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:02.317204 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:02.317204 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:02.317204 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:02.317204 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:02.317204 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:02.317204 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:02.317204 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:02.317204 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:02.317204 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:02.317204 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:02.317204 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:02.317204 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:02.317204 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:02.317204 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:02.317204 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:02.317204 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:02.318136 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:02.318136 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:02.318136 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:02.318136 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:02.318136 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:02.318136 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:02.318136 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:02.318136 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:02.318136 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:02.318136 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:02.318136 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 12 17:19:02.318136 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:02.318136 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 41/85] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:02 2022] GET /identity/v3/roles/member => generated 84 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:02.319980 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82765df-19ef-4ba5-9241-9466aba93b82 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:02.333612 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7551e108-ddea-4b76-b4b7-bb9beddeb299 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:02.335079 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7551e108-ddea-4b76-b4b7-bb9beddeb299 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:02.335541 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7551e108-ddea-4b76-b4b7-bb9beddeb299 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:02.336021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7551e108-ddea-4b76-b4b7-bb9beddeb299 None admin] PATH_INFO: `/v3/users/dbc90e3cb55f431e91cd23da7b8958f2` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:02.337511 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7551e108-ddea-4b76-b4b7-bb9beddeb299 None admin] RBAC: Authorizing `identity:get_user(user_id=dbc90e3cb55f431e91cd23da7b8958f2)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:02.339262 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7551e108-ddea-4b76-b4b7-bb9beddeb299 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:02.341195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 45/86] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:19:02 2022] GET /identity/v3/users/dbc90e3cb55f431e91cd23da7b8958f2 => generated 275 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:02.342248 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7551e108-ddea-4b76-b4b7-bb9beddeb299 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:02.436474 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:02.439396 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82765df-19ef-4ba5-9241-9466aba93b82 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:02.456254 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7551e108-ddea-4b76-b4b7-bb9beddeb299 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:02.478871 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adcbe0a7-a37f-4197-ad83-1c4b1106ffa8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:02.479862 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcbe0a7-a37f-4197-ad83-1c4b1106ffa8 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:02.480253 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcbe0a7-a37f-4197-ad83-1c4b1106ffa8 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:02.480650 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcbe0a7-a37f-4197-ad83-1c4b1106ffa8 None admin] PATH_INFO: `/v3/projects/4857259d2a364dff960547dc0aac75e0` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:02.481996 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adcbe0a7-a37f-4197-ad83-1c4b1106ffa8 None admin] RBAC: Authorizing `identity:get_project(project_id=4857259d2a364dff960547dc0aac75e0)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:02.483902 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adcbe0a7-a37f-4197-ad83-1c4b1106ffa8 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:02.485830 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 42/87] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:02 2022] GET /identity/v3/projects/4857259d2a364dff960547dc0aac75e0 => generated 298 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:02.486860 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adcbe0a7-a37f-4197-ad83-1c4b1106ffa8 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:02.495504 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14ceb28-cec9-49e6-a292-ec9f00e7f510 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:02.496473 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14ceb28-cec9-49e6-a292-ec9f00e7f510 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:02.497012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14ceb28-cec9-49e6-a292-ec9f00e7f510 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:02.497485 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14ceb28-cec9-49e6-a292-ec9f00e7f510 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:02.498766 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b14ceb28-cec9-49e6-a292-ec9f00e7f510 None admin] RBAC: Adding query filter params (role.id=4c5d52f873f94dc1b10918068376add5, scope.project.id=4857259d2a364dff960547dc0aac75e0, user.id=dbc90e3cb55f431e91cd23da7b8958f2) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:02.499283 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b14ceb28-cec9-49e6-a292-ec9f00e7f510 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:02.501288 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b14ceb28-cec9-49e6-a292-ec9f00e7f510 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:02.508244 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 46/88] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:19:02 2022] GET /identity/v3/role_assignments?user.id=dbc90e3cb55f431e91cd23da7b8958f2&scope.project.id=4857259d2a364dff960547dc0aac75e0&role.id=4c5d52f873f94dc1b10918068376add5 => generated 259 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:02.509319 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14ceb28-cec9-49e6-a292-ec9f00e7f510 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:02.519695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:02.633858 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14ceb28-cec9-49e6-a292-ec9f00e7f510 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:02.673195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96b00c5-37b6-480f-932d-a4e33908f958 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:02.674244 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96b00c5-37b6-480f-932d-a4e33908f958 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:02.674790 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96b00c5-37b6-480f-932d-a4e33908f958 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:02.675270 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96b00c5-37b6-480f-932d-a4e33908f958 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:02.676449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adcbe0a7-a37f-4197-ad83-1c4b1106ffa8 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:02.746765 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0322f7f3-10ce-4aba-b560-139b047e8f7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:02.747461 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0322f7f3-10ce-4aba-b560-139b047e8f7b None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:02.747758 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0322f7f3-10ce-4aba-b560-139b047e8f7b None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:02.748047 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0322f7f3-10ce-4aba-b560-139b047e8f7b None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:02.748830 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0322f7f3-10ce-4aba-b560-139b047e8f7b None admin] RBAC: Adding query filter params (name=member) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:02.749310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0322f7f3-10ce-4aba-b560-139b047e8f7b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:02.752432 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0322f7f3-10ce-4aba-b560-139b047e8f7b None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:02.780036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 43/89] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Jul 12 17:19:02 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:02.781028 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0322f7f3-10ce-4aba-b560-139b047e8f7b None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:02.805595 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96b00c5-37b6-480f-932d-a4e33908f958 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:02.807888 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96b00c5-37b6-480f-932d-a4e33908f958 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:02.811715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:02.822623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:02.916865 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 47/90] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:19:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 408 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 12 17:19:02.918133 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96b00c5-37b6-480f-932d-a4e33908f958 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:02.980418 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0322f7f3-10ce-4aba-b560-139b047e8f7b None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:03.032877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96b00c5-37b6-480f-932d-a4e33908f958 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:03.036524 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:03.049589 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54944945-c97c-4e44-b238-e497558f9b18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:03.050265 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54944945-c97c-4e44-b238-e497558f9b18 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:03.050980 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54944945-c97c-4e44-b238-e497558f9b18 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:03.050980 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54944945-c97c-4e44-b238-e497558f9b18 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:03.051836 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54944945-c97c-4e44-b238-e497558f9b18 None admin] RBAC: Adding query filter params (role.id=eb05bf286c894ea9bf44a6c023a4323b, scope.project.id=4857259d2a364dff960547dc0aac75e0, user.id=df85f597d0ba471381a5a8f9b6886705) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:03.052169 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54944945-c97c-4e44-b238-e497558f9b18 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:03.053742 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54944945-c97c-4e44-b238-e497558f9b18 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:03.059972 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 44/91] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:19:02 2022] GET /identity/v3/role_assignments?user.id=df85f597d0ba471381a5a8f9b6886705&scope.project.id=4857259d2a364dff960547dc0aac75e0&role.id=eb05bf286c894ea9bf44a6c023a4323b => generated 259 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:03.060779 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54944945-c97c-4e44-b238-e497558f9b18 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:03.075383 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e547d40-6e97-4cc8-8f92-f298a9d1b447 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:03.076688 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e547d40-6e97-4cc8-8f92-f298a9d1b447 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:03.077194 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e547d40-6e97-4cc8-8f92-f298a9d1b447 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:03.077680 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e547d40-6e97-4cc8-8f92-f298a9d1b447 None admin] PATH_INFO: `/v3/users/dbc90e3cb55f431e91cd23da7b8958f2` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:03.079206 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e547d40-6e97-4cc8-8f92-f298a9d1b447 None admin] RBAC: Authorizing `identity:get_user(user_id=dbc90e3cb55f431e91cd23da7b8958f2)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:03.080915 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e547d40-6e97-4cc8-8f92-f298a9d1b447 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:03.088301 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 48/92] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:19:02 2022] GET /identity/v3/users/dbc90e3cb55f431e91cd23da7b8958f2 => generated 275 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:03.090187 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e547d40-6e97-4cc8-8f92-f298a9d1b447 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:03.204661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e547d40-6e97-4cc8-8f92-f298a9d1b447 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:03.223468 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:03.239064 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54944945-c97c-4e44-b238-e497558f9b18 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:03.250941 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6108dc20-884a-42c2-8bad-2519606f5308 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:03.252005 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6108dc20-884a-42c2-8bad-2519606f5308 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:03.252503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6108dc20-884a-42c2-8bad-2519606f5308 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:03.253015 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6108dc20-884a-42c2-8bad-2519606f5308 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:03.300133 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:03.308285 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ddda72b-1d75-4834-acab-07696da2dcf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:03.309183 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ddda72b-1d75-4834-acab-07696da2dcf1 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:03.309512 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ddda72b-1d75-4834-acab-07696da2dcf1 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:03.309788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ddda72b-1d75-4834-acab-07696da2dcf1 None admin] PATH_INFO: `/v3/projects/4857259d2a364dff960547dc0aac75e0` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:03.315425 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ddda72b-1d75-4834-acab-07696da2dcf1 None admin] RBAC: Authorizing `identity:get_project(project_id=4857259d2a364dff960547dc0aac75e0)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:03.316914 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ddda72b-1d75-4834-acab-07696da2dcf1 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:03.322036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 45/93] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:03 2022] GET /identity/v3/projects/4857259d2a364dff960547dc0aac75e0 => generated 298 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:03.326467 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ddda72b-1d75-4834-acab-07696da2dcf1 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:03.343854 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6108dc20-884a-42c2-8bad-2519606f5308 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:03.345836 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6108dc20-884a-42c2-8bad-2519606f5308 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:03.489623 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 49/94] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:19:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 400 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 12 17:19:03.492040 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ddda72b-1d75-4834-acab-07696da2dcf1 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:03.495463 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6108dc20-884a-42c2-8bad-2519606f5308 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:03.563248 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cda5e7c-1b99-4d67-96ff-393567d84f1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:03.563846 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cda5e7c-1b99-4d67-96ff-393567d84f1f None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:03.566464 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cda5e7c-1b99-4d67-96ff-393567d84f1f None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:03.566464 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cda5e7c-1b99-4d67-96ff-393567d84f1f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:03.566464 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cda5e7c-1b99-4d67-96ff-393567d84f1f None admin] RBAC: Adding query filter params (role.id=0c239663092d459e8c9724963e78da76, scope.project.id=4857259d2a364dff960547dc0aac75e0, user.id=dbc90e3cb55f431e91cd23da7b8958f2) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:03.566464 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cda5e7c-1b99-4d67-96ff-393567d84f1f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:03.567201 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cda5e7c-1b99-4d67-96ff-393567d84f1f None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:03.585948 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 46/95] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:19:03 2022] GET /identity/v3/role_assignments?user.id=dbc90e3cb55f431e91cd23da7b8958f2&scope.project.id=4857259d2a364dff960547dc0aac75e0&role.id=0c239663092d459e8c9724963e78da76 => generated 259 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:19:03.635537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:03.686520 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6108dc20-884a-42c2-8bad-2519606f5308 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:03.739188 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f66fec-1cab-4101-b231-05f3e7d79710 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:03.740631 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f66fec-1cab-4101-b231-05f3e7d79710 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:03.740925 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f66fec-1cab-4101-b231-05f3e7d79710 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:03.741207 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f66fec-1cab-4101-b231-05f3e7d79710 None admin] PATH_INFO: `/v3/projects/bdf9abb6d70a41acbad1d4060c21408f` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:03.742711 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f66fec-1cab-4101-b231-05f3e7d79710 None admin] RBAC: Authorizing `identity:get_project(project_id=bdf9abb6d70a41acbad1d4060c21408f)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:03.744196 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f66fec-1cab-4101-b231-05f3e7d79710 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:03.745984 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 50/96] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:03 2022] GET /identity/v3/projects/bdf9abb6d70a41acbad1d4060c21408f => generated 312 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:03.752262 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cda5e7c-1b99-4d67-96ff-393567d84f1f None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:03.806148 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:03.899870 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cda5e7c-1b99-4d67-96ff-393567d84f1f None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:03.950076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423e911e-28aa-4a99-a87c-1523ba5a29e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:03.950798 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423e911e-28aa-4a99-a87c-1523ba5a29e5 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:03.951095 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423e911e-28aa-4a99-a87c-1523ba5a29e5 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:03.951411 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423e911e-28aa-4a99-a87c-1523ba5a29e5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:03.952323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423e911e-28aa-4a99-a87c-1523ba5a29e5 None admin] RBAC: Adding query filter params (role.id=4c5d52f873f94dc1b10918068376add5, scope.project.id=bdf9abb6d70a41acbad1d4060c21408f, user.id=dbc90e3cb55f431e91cd23da7b8958f2) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:03.952649 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423e911e-28aa-4a99-a87c-1523ba5a29e5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:03.954157 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-423e911e-28aa-4a99-a87c-1523ba5a29e5 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:03.963182 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 47/97] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:19:03 2022] GET /identity/v3/role_assignments?user.id=dbc90e3cb55f431e91cd23da7b8958f2&scope.project.id=bdf9abb6d70a41acbad1d4060c21408f&role.id=4c5d52f873f94dc1b10918068376add5 => generated 259 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:04.293003 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67134e8-5edd-468d-b98f-db73181085dd None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:04.293384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67134e8-5edd-468d-b98f-db73181085dd None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:04.293605 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67134e8-5edd-468d-b98f-db73181085dd None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:04.295045 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 51/98] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:04.300169 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae11e8e-b426-4b02-b501-8618702628f4 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:04.300471 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae11e8e-b426-4b02-b501-8618702628f4 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:04.300765 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae11e8e-b426-4b02-b501-8618702628f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:04.331493 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-aae11e8e-b426-4b02-b501-8618702628f4 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:04.332702 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-aae11e8e-b426-4b02-b501-8618702628f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:04.422337 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 48/99] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 12 17:19:04.458631 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67134e8-5edd-468d-b98f-db73181085dd None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:04.495206 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2758fb78-104a-436b-80f4-bcf2103a8982 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:04.495527 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2758fb78-104a-436b-80f4-bcf2103a8982 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:04.495775 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2758fb78-104a-436b-80f4-bcf2103a8982 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:04.497166 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 49/100] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:04.503092 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10bdee7-d6ff-44f7-a2f7-15817d72e699 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:04.503386 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10bdee7-d6ff-44f7-a2f7-15817d72e699 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:04.503673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10bdee7-d6ff-44f7-a2f7-15817d72e699 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:04.533321 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-b10bdee7-d6ff-44f7-a2f7-15817d72e699 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:04.534613 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-b10bdee7-d6ff-44f7-a2f7-15817d72e699 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:04.564856 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:04.655267 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 50/101] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 12 17:19:04.688119 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67134e8-5edd-468d-b98f-db73181085dd None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:04.691922 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10bdee7-d6ff-44f7-a2f7-15817d72e699 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:04.771804 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b9b1dcd-7d7d-48df-a356-d62e81c9bd54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:04.773033 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9b1dcd-7d7d-48df-a356-d62e81c9bd54 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:04.773538 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9b1dcd-7d7d-48df-a356-d62e81c9bd54 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:04.774025 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9b1dcd-7d7d-48df-a356-d62e81c9bd54 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:04.801156 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:04.811035 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b9b1dcd-7d7d-48df-a356-d62e81c9bd54 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:04.819874 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-7b9b1dcd-7d7d-48df-a356-d62e81c9bd54 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 12 17:19:04.819874 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:04.819874 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:04.819874 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:04.819874 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:04.819874 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:04.819874 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:04.819874 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:04.819874 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:04.819874 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:04.819874 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:04.819874 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:04.819874 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:04.819874 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:04.819874 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:04.819874 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:04.820726 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:04.820726 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:04.820726 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:04.820726 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:04.820726 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:04.820726 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:04.820726 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:04.820726 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:04.820726 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:04.820726 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:04.820726 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:04.820726 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:04.820726 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:04.820726 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:04.820726 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:04.820726 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:04.821504 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:04.821504 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:04.821504 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:04.821504 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:04.821504 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:04.821504 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:04.821504 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:04.821504 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:04.821504 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:04.821504 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:04.821504 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 12 17:19:04.821504 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:19:04.824020 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 52/102] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:04 2022] GET /identity/v3/roles/reader => generated 84 bytes in 366 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:04.828887 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10bdee7-d6ff-44f7-a2f7-15817d72e699 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:04.829999 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b9b1dcd-7d7d-48df-a356-d62e81c9bd54 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:04.863595 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c3680e-7b0b-485a-ac72-796c6a1177f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:04.864510 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c3680e-7b0b-485a-ac72-796c6a1177f4 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:04.865655 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c3680e-7b0b-485a-ac72-796c6a1177f4 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:04.866357 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c3680e-7b0b-485a-ac72-796c6a1177f4 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:04.867782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c3680e-7b0b-485a-ac72-796c6a1177f4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:04.869508 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c3680e-7b0b-485a-ac72-796c6a1177f4 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:04.871439 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 51/103] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:19:04 2022] GET /identity/v3/domains/default => generated 199 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:04.879460 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c3680e-7b0b-485a-ac72-796c6a1177f4 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:04.907677 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:05.050750 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c3680e-7b0b-485a-ac72-796c6a1177f4 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:05.112754 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b9b1dcd-7d7d-48df-a356-d62e81c9bd54 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:05.132332 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bca6d34-50ed-446c-8124-ec0d4bd0b7eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:05.133485 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bca6d34-50ed-446c-8124-ec0d4bd0b7eb None admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:05.133790 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bca6d34-50ed-446c-8124-ec0d4bd0b7eb None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:05.134081 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bca6d34-50ed-446c-8124-ec0d4bd0b7eb None admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:05.135189 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bca6d34-50ed-446c-8124-ec0d4bd0b7eb None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:05.136884 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bca6d34-50ed-446c-8124-ec0d4bd0b7eb None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:05.204096 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c957f9-da4c-4dbe-ad98-d83ac7eb7ac2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:05.204796 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c957f9-da4c-4dbe-ad98-d83ac7eb7ac2 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:05.205093 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c957f9-da4c-4dbe-ad98-d83ac7eb7ac2 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:05.205382 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c957f9-da4c-4dbe-ad98-d83ac7eb7ac2 None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:05.206211 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c957f9-da4c-4dbe-ad98-d83ac7eb7ac2 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:05.206571 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c957f9-da4c-4dbe-ad98-d83ac7eb7ac2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:05.207980 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c957f9-da4c-4dbe-ad98-d83ac7eb7ac2 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:05.225765 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 53/104] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Jul 12 17:19:04 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 396 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:05.230979 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c957f9-da4c-4dbe-ad98-d83ac7eb7ac2 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:05.255114 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 52/105] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Jul 12 17:19:04 2022] POST /identity/v3/projects => generated 302 bytes in 376 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:19:05.328359 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:05.338538 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb710cd-4c46-4287-a254-a4d5985537bc None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:05.339047 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb710cd-4c46-4287-a254-a4d5985537bc None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:05.339155 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdb710cd-4c46-4287-a254-a4d5985537bc None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:05.340546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 53/106] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:05.346332 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e45df6-4921-4218-9588-dd98eda2db2e None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:05.346948 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e45df6-4921-4218-9588-dd98eda2db2e None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:05.347429 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e45df6-4921-4218-9588-dd98eda2db2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:05.383567 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-f3e45df6-4921-4218-9588-dd98eda2db2e None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:05.385156 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-f3e45df6-4921-4218-9588-dd98eda2db2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:05.395898 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c957f9-da4c-4dbe-ad98-d83ac7eb7ac2 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:05.473493 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6427af55-f7ad-4f6e-a8a7-4bc149e406f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:05.475114 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6427af55-f7ad-4f6e-a8a7-4bc149e406f0 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:05.475504 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6427af55-f7ad-4f6e-a8a7-4bc149e406f0 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:05.475900 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6427af55-f7ad-4f6e-a8a7-4bc149e406f0 None admin] PATH_INFO: `/v3/users/cf24bc418fc94f0f974e1870bd9bea31` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:05.496044 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6427af55-f7ad-4f6e-a8a7-4bc149e406f0 None admin] RBAC: Authorizing `identity:get_user(user_id=cf24bc418fc94f0f974e1870bd9bea31)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:05.497812 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6427af55-f7ad-4f6e-a8a7-4bc149e406f0 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:05.500370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 54/107] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:19:05 2022] GET /identity/v3/users/cf24bc418fc94f0f974e1870bd9bea31 => generated 289 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:19:05.508077 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6427af55-f7ad-4f6e-a8a7-4bc149e406f0 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:05.569420 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:05.578950 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 54/108] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 12 17:19:05.620643 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e45df6-4921-4218-9588-dd98eda2db2e None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:05.645289 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6427af55-f7ad-4f6e-a8a7-4bc149e406f0 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:05.693988 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:05.720889 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aeb4722-6719-442b-8217-56b3c1f42f4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:05.721912 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aeb4722-6719-442b-8217-56b3c1f42f4a None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:05.723487 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aeb4722-6719-442b-8217-56b3c1f42f4a None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:05.725822 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aeb4722-6719-442b-8217-56b3c1f42f4a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:05.830466 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e45df6-4921-4218-9588-dd98eda2db2e None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:05.857518 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aeb4722-6719-442b-8217-56b3c1f42f4a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:05.859610 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aeb4722-6719-442b-8217-56b3c1f42f4a None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:05.890627 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf086a3-799e-4143-8d5c-092cd3337622 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:05.891460 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf086a3-799e-4143-8d5c-092cd3337622 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:05.891755 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf086a3-799e-4143-8d5c-092cd3337622 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:05.892042 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf086a3-799e-4143-8d5c-092cd3337622 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:05.916348 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cf086a3-799e-4143-8d5c-092cd3337622 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:05.932533 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-9cf086a3-799e-4143-8d5c-092cd3337622 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 12 17:19:05.932533 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:05.932533 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:05.932533 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:05.932533 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:05.932533 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:05.932533 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:05.932533 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:05.932533 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:05.932533 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:05.932533 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:05.932533 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:05.932533 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:05.932533 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:05.932533 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:05.932533 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:05.936367 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:05.936367 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:05.936367 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:05.936367 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:05.936367 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:05.936367 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:05.936367 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:05.936367 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:05.936367 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:05.936367 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:05.936367 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:05.936367 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:05.936367 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:05.936367 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:05.936367 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:05.936367 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:05.937169 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:05.937169 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:05.937169 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:05.937169 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:05.937169 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:05.937169 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:05.937169 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:05.937169 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:05.937169 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:05.937169 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:05.937169 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 12 17:19:05.937169 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:05.937169 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 55/109] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:05 2022] GET /identity/v3/roles/member => generated 84 bytes in 314 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:19:05.939738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf086a3-799e-4143-8d5c-092cd3337622 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:05.986915 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 55/110] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:19:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 479 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 12 17:19:05.995696 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aeb4722-6719-442b-8217-56b3c1f42f4a None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:06.025455 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:06.079532 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:06.116207 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aeb4722-6719-442b-8217-56b3c1f42f4a None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:06.153971 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50adf82d-7084-4d12-84b9-2d3461713bcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:06.155280 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50adf82d-7084-4d12-84b9-2d3461713bcd None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:06.155827 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50adf82d-7084-4d12-84b9-2d3461713bcd None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:06.156306 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50adf82d-7084-4d12-84b9-2d3461713bcd None admin] PATH_INFO: `/v3/projects/4857259d2a364dff960547dc0aac75e0` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:06.157949 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50adf82d-7084-4d12-84b9-2d3461713bcd None admin] RBAC: Authorizing `identity:get_project(project_id=4857259d2a364dff960547dc0aac75e0)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:06.159686 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50adf82d-7084-4d12-84b9-2d3461713bcd None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:06.161710 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 56/111] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:05 2022] GET /identity/v3/projects/4857259d2a364dff960547dc0aac75e0 => generated 298 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:19:06.168657 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50adf82d-7084-4d12-84b9-2d3461713bcd None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:06.184552 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf086a3-799e-4143-8d5c-092cd3337622 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:06.252502 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58465924-5136-4fbf-bb18-1ad90a91a520 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:06.253780 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58465924-5136-4fbf-bb18-1ad90a91a520 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:06.254666 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58465924-5136-4fbf-bb18-1ad90a91a520 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:06.255175 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58465924-5136-4fbf-bb18-1ad90a91a520 None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:06.256545 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58465924-5136-4fbf-bb18-1ad90a91a520 None admin] RBAC: Adding query filter params (name=member) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:06.257668 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58465924-5136-4fbf-bb18-1ad90a91a520 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:06.259563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58465924-5136-4fbf-bb18-1ad90a91a520 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:06.273388 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 56/112] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Jul 12 17:19:05 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 334 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:19:06.275376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1419afb-d7b1-431d-8608-531709338c02 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:06.275878 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1419afb-d7b1-431d-8608-531709338c02 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:06.276383 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1419afb-d7b1-431d-8608-531709338c02 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:06.277848 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 57/113] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:06.284278 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1419afb-d7b1-431d-8608-531709338c02 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:06.337303 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:06.351046 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50adf82d-7084-4d12-84b9-2d3461713bcd None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:06.410382 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1419afb-d7b1-431d-8608-531709338c02 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:06.437444 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:06.450881 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37fb0c1b-4c19-40d7-9c9d-b3034fc6f5b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:06.451166 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a838a0-6b8e-479f-847f-01d6f5547101 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:06.451765 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fb0c1b-4c19-40d7-9c9d-b3034fc6f5b0 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:06.453955 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a838a0-6b8e-479f-847f-01d6f5547101 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:06.454071 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fb0c1b-4c19-40d7-9c9d-b3034fc6f5b0 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:06.454159 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a838a0-6b8e-479f-847f-01d6f5547101 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:06.454241 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fb0c1b-4c19-40d7-9c9d-b3034fc6f5b0 None admin] PATH_INFO: `/v3/users/dbc90e3cb55f431e91cd23da7b8958f2` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:06.454338 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a838a0-6b8e-479f-847f-01d6f5547101 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:06.454338 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a838a0-6b8e-479f-847f-01d6f5547101 None admin] RBAC: Adding query filter params (role.id=6670fba31f504efb9ae95813d9b6d660, scope.project.id=4857259d2a364dff960547dc0aac75e0, user.id=cf24bc418fc94f0f974e1870bd9bea31) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:06.454338 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a838a0-6b8e-479f-847f-01d6f5547101 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:06.454596 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37fb0c1b-4c19-40d7-9c9d-b3034fc6f5b0 None admin] RBAC: Authorizing `identity:get_user(user_id=dbc90e3cb55f431e91cd23da7b8958f2)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:06.455069 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1a838a0-6b8e-479f-847f-01d6f5547101 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:06.456838 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37fb0c1b-4c19-40d7-9c9d-b3034fc6f5b0 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:06.460681 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 58/114] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:19:06 2022] GET /identity/v3/users/dbc90e3cb55f431e91cd23da7b8958f2 => generated 275 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:06.462636 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76cb7968-66fd-401f-afaf-05a02fbc33f1 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:06.463176 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76cb7968-66fd-401f-afaf-05a02fbc33f1 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:06.463604 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76cb7968-66fd-401f-afaf-05a02fbc33f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:06.469545 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 57/115] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:19:06 2022] GET /identity/v3/role_assignments?user.id=cf24bc418fc94f0f974e1870bd9bea31&scope.project.id=4857259d2a364dff960547dc0aac75e0&role.id=6670fba31f504efb9ae95813d9b6d660 => generated 259 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:06.470750 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a838a0-6b8e-479f-847f-01d6f5547101 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:06.483885 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-76cb7968-66fd-401f-afaf-05a02fbc33f1 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:06.485437 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-76cb7968-66fd-401f-afaf-05a02fbc33f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:06.657811 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 59/116] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:06.688542 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1a838a0-6b8e-479f-847f-01d6f5547101 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:06.695150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76cb7968-66fd-401f-afaf-05a02fbc33f1 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:06.746337 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4af0a3a-20d0-4c9d-83e0-970894ebdd97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:06.747771 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4af0a3a-20d0-4c9d-83e0-970894ebdd97 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:06.748245 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4af0a3a-20d0-4c9d-83e0-970894ebdd97 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:06.748799 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4af0a3a-20d0-4c9d-83e0-970894ebdd97 None admin] PATH_INFO: `/v3/projects/4857259d2a364dff960547dc0aac75e0` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:06.750495 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4af0a3a-20d0-4c9d-83e0-970894ebdd97 None admin] RBAC: Authorizing `identity:get_project(project_id=4857259d2a364dff960547dc0aac75e0)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:06.752137 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4af0a3a-20d0-4c9d-83e0-970894ebdd97 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:06.754530 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 58/117] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:06 2022] GET /identity/v3/projects/4857259d2a364dff960547dc0aac75e0 => generated 298 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:06.766299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4af0a3a-20d0-4c9d-83e0-970894ebdd97 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:06.801971 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:06.876059 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76cb7968-66fd-401f-afaf-05a02fbc33f1 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:06.900028 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:06.916470 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0463919c-4bc1-4224-8dd3-d687ead8f00c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:06.917280 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0463919c-4bc1-4224-8dd3-d687ead8f00c None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:06.917573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0463919c-4bc1-4224-8dd3-d687ead8f00c None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:06.917857 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0463919c-4bc1-4224-8dd3-d687ead8f00c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:06.935707 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4af0a3a-20d0-4c9d-83e0-970894ebdd97 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:06.942682 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0463919c-4bc1-4224-8dd3-d687ead8f00c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:06.959573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-0463919c-4bc1-4224-8dd3-d687ead8f00c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:19:06.959573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:06.959573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:06.959573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:06.959573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:06.959573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:06.959573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:06.959573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:06.959573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:06.959573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:06.959573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:06.959573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:06.959573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:06.959573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:06.959573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:06.959573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:06.960525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:06.960525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:06.960525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:06.960525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:06.960525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:06.960525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:06.960525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:06.960525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:06.960525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:06.960525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:06.960525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:06.960525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:06.960525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:06.960525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:06.960525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:06.960525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:06.961281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:06.961281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:06.961281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:06.961281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:06.961281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:06.961281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:06.961281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:06.961281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:06.961281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:06.961281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:06.961281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:19:06.961281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:06.965847 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 60/118] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Jul 12 17:19:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:19:06.967230 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c731fd98-df18-407e-b671-30b12edda95c None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:06.967528 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c731fd98-df18-407e-b671-30b12edda95c None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:06.967817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c731fd98-df18-407e-b671-30b12edda95c None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:06.979431 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 61/119] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:06 2022] GET /identity => generated 270 bytes in 13 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:06.980320 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c731fd98-df18-407e-b671-30b12edda95c None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:07.010070 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072b8d67-115f-46f5-85c4-d9adfaa159ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:07.011122 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072b8d67-115f-46f5-85c4-d9adfaa159ba None admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:07.011415 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072b8d67-115f-46f5-85c4-d9adfaa159ba None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:07.011788 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072b8d67-115f-46f5-85c4-d9adfaa159ba None admin] PATH_INFO: `/v3/projects/4857259d2a364dff960547dc0aac75e0/users/dbc90e3cb55f431e91cd23da7b8958f2/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:07.014275 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-072b8d67-115f-46f5-85c4-d9adfaa159ba None admin] RBAC: Authorizing `identity:create_grant(project_id=4857259d2a364dff960547dc0aac75e0, user_id=dbc90e3cb55f431e91cd23da7b8958f2, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:07.016543 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-072b8d67-115f-46f5-85c4-d9adfaa159ba None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:07.080509 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 59/120] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Jul 12 17:19:06 2022] PUT /identity/v3/projects/4857259d2a364dff960547dc0aac75e0/users/dbc90e3cb55f431e91cd23da7b8958f2/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 315 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:19:07.081876 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6693c2-8b7a-4cf7-9924-f06f49090e2f None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:07.082184 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6693c2-8b7a-4cf7-9924-f06f49090e2f None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:07.082572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6693c2-8b7a-4cf7-9924-f06f49090e2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:07.101760 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-0c6693c2-8b7a-4cf7-9924-f06f49090e2f None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:07.103075 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-0c6693c2-8b7a-4cf7-9924-f06f49090e2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:07.107687 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c731fd98-df18-407e-b671-30b12edda95c None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:07.157861 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c762731-86b4-4354-b572-86e040c130e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:07.158860 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c762731-86b4-4354-b572-86e040c130e5 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:07.159155 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c762731-86b4-4354-b572-86e040c130e5 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:07.159440 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c762731-86b4-4354-b572-86e040c130e5 None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:07.160546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c762731-86b4-4354-b572-86e040c130e5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:07.161293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c762731-86b4-4354-b572-86e040c130e5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:07.163826 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c762731-86b4-4354-b572-86e040c130e5 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:07.178157 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 62/121] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:07.179605 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f692c4-802e-436d-8f00-73a8cd6a922f None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:07.179958 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 60/122] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:07.180129 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f692c4-802e-436d-8f00-73a8cd6a922f None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:07.180435 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f692c4-802e-436d-8f00-73a8cd6a922f None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:07.181997 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 63/123] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:07.184990 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6693c2-8b7a-4cf7-9924-f06f49090e2f None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:07.187683 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f43c6e-1416-4412-80c4-ee649a5e1ce7 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:07.187984 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f43c6e-1416-4412-80c4-ee649a5e1ce7 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:07.188270 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f43c6e-1416-4412-80c4-ee649a5e1ce7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:07.208311 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:07.211327 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-03f43c6e-1416-4412-80c4-ee649a5e1ce7 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:07.212733 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-03f43c6e-1416-4412-80c4-ee649a5e1ce7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:07.278616 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:07.330732 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 64/124] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 12 17:19:07.331707 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f43c6e-1416-4412-80c4-ee649a5e1ce7 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:07.379068 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6693c2-8b7a-4cf7-9924-f06f49090e2f None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:07.429520 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c851de-8e1a-4de7-a37d-fc0cad2761dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:07.430712 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c851de-8e1a-4de7-a37d-fc0cad2761dd None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:07.431004 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c851de-8e1a-4de7-a37d-fc0cad2761dd None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:07.431286 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c851de-8e1a-4de7-a37d-fc0cad2761dd None admin] PATH_INFO: `/v3/users/df85f597d0ba471381a5a8f9b6886705` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:07.433150 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c851de-8e1a-4de7-a37d-fc0cad2761dd None admin] RBAC: Authorizing `identity:get_user(user_id=df85f597d0ba471381a5a8f9b6886705)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:07.434629 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c851de-8e1a-4de7-a37d-fc0cad2761dd None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:07.437021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 61/125] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:19:07 2022] GET /identity/v3/users/df85f597d0ba471381a5a8f9b6886705 => generated 247 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:07.439280 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c851de-8e1a-4de7-a37d-fc0cad2761dd None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:07.456147 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f43c6e-1416-4412-80c4-ee649a5e1ce7 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:07.494329 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72982664-4c9e-48e3-ac49-d7bdb3558461 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:07.495307 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72982664-4c9e-48e3-ac49-d7bdb3558461 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:07.495689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72982664-4c9e-48e3-ac49-d7bdb3558461 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:07.496046 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72982664-4c9e-48e3-ac49-d7bdb3558461 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:07.518205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72982664-4c9e-48e3-ac49-d7bdb3558461 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:07.535631 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:07.535794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-72982664-4c9e-48e3-ac49-d7bdb3558461 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 12 17:19:07.535794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:07.535794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:07.535794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:07.535794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:07.535794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:07.535794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:07.535794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:07.535794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:07.535794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:07.535794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:07.535794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:07.535794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:07.535794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:07.535794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:07.535794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:07.536671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:07.536671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:07.536671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:07.536671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:07.536671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:07.536671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:07.536671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:07.536671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:07.536671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:07.536671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:07.536671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:07.536671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:07.536671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:07.536671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:07.536671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:07.536671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:07.537996 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:07.537996 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:07.537996 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:07.537996 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:07.537996 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:07.537996 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:07.537996 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:07.537996 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:07.537996 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:07.537996 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:07.537996 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 12 17:19:07.537996 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:07.537996 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 65/126] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Jul 12 17:19:07 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:07.538778 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72982664-4c9e-48e3-ac49-d7bdb3558461 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:07.561516 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c851de-8e1a-4de7-a37d-fc0cad2761dd None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:07.612295 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d601d0b-80e0-4702-bac7-e20fc9e0e9c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:07.613094 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d601d0b-80e0-4702-bac7-e20fc9e0e9c2 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:07.613378 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d601d0b-80e0-4702-bac7-e20fc9e0e9c2 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:07.613654 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d601d0b-80e0-4702-bac7-e20fc9e0e9c2 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:07.625738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d601d0b-80e0-4702-bac7-e20fc9e0e9c2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:07.627452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:07.639877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-0d601d0b-80e0-4702-bac7-e20fc9e0e9c2 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 12 17:19:07.639877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:07.639877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:07.639877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:07.639877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:07.639877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:07.639877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:07.639877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:07.639877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:07.639877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:07.639877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:07.639877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:07.639877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:07.639877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:07.639877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:07.639877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:07.642111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:07.642111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:07.642111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:07.642111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:07.642111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:07.642111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:07.642111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:07.642111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:07.642111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:07.642111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:07.642111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:07.642111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:07.642111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:07.642111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:07.642111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:07.642111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:07.644193 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:07.644193 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:07.644193 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:07.644193 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:07.644193 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:07.644193 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:07.644193 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:07.644193 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:07.644193 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:07.644193 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:07.644193 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 12 17:19:07.644193 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:19:07.644193 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 62/127] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:07 2022] GET /identity/v3/roles/member => generated 84 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:07.644193 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d601d0b-80e0-4702-bac7-e20fc9e0e9c2 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:07.751916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d601d0b-80e0-4702-bac7-e20fc9e0e9c2 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:07.772438 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72982664-4c9e-48e3-ac49-d7bdb3558461 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:07.788300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2bd4ed-9ce5-483a-99a8-bc16cf89dd37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:07.788978 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2bd4ed-9ce5-483a-99a8-bc16cf89dd37 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:07.789267 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2bd4ed-9ce5-483a-99a8-bc16cf89dd37 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:07.789552 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2bd4ed-9ce5-483a-99a8-bc16cf89dd37 None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:07.790449 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b2bd4ed-9ce5-483a-99a8-bc16cf89dd37 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:07.790775 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b2bd4ed-9ce5-483a-99a8-bc16cf89dd37 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:07.792155 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b2bd4ed-9ce5-483a-99a8-bc16cf89dd37 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:07.797674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:07.802313 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 63/128] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:19:07 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:07.803281 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2bd4ed-9ce5-483a-99a8-bc16cf89dd37 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:07.871158 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b841ca4-390d-4311-bd0c-58d7ae041158 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:07.872429 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b841ca4-390d-4311-bd0c-58d7ae041158 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:07.872926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b841ca4-390d-4311-bd0c-58d7ae041158 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:07.873425 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b841ca4-390d-4311-bd0c-58d7ae041158 None admin] PATH_INFO: `/v3/projects/4857259d2a364dff960547dc0aac75e0` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:07.879311 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b841ca4-390d-4311-bd0c-58d7ae041158 None admin] RBAC: Authorizing `identity:get_project(project_id=4857259d2a364dff960547dc0aac75e0)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:07.882277 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b841ca4-390d-4311-bd0c-58d7ae041158 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:07.884346 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 66/129] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:07 2022] GET /identity/v3/projects/4857259d2a364dff960547dc0aac75e0 => generated 298 bytes in 346 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:07.885344 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b841ca4-390d-4311-bd0c-58d7ae041158 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:07.928362 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2bd4ed-9ce5-483a-99a8-bc16cf89dd37 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:07.997180 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cce2eb1-1014-4a9b-97d8-51dee2d526e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:07.997858 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cce2eb1-1014-4a9b-97d8-51dee2d526e7 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:07.998144 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cce2eb1-1014-4a9b-97d8-51dee2d526e7 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:08.000544 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cce2eb1-1014-4a9b-97d8-51dee2d526e7 None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:08.001327 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cce2eb1-1014-4a9b-97d8-51dee2d526e7 None admin] RBAC: Adding query filter params (name=member) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:08.001645 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cce2eb1-1014-4a9b-97d8-51dee2d526e7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:08.003139 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cce2eb1-1014-4a9b-97d8-51dee2d526e7 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:08.022556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 64/130] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Jul 12 17:19:07 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:08.023400 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cce2eb1-1014-4a9b-97d8-51dee2d526e7 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:08.048434 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:08.051959 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b841ca4-390d-4311-bd0c-58d7ae041158 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:08.090410 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfad1b4-6220-4ef3-a588-5c9bfc5a6508 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:08.092088 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfad1b4-6220-4ef3-a588-5c9bfc5a6508 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:08.092379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfad1b4-6220-4ef3-a588-5c9bfc5a6508 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:08.092664 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dfad1b4-6220-4ef3-a588-5c9bfc5a6508 None admin] PATH_INFO: `/v3/users/dbc90e3cb55f431e91cd23da7b8958f2` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:08.094755 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dfad1b4-6220-4ef3-a588-5c9bfc5a6508 None admin] RBAC: Authorizing `identity:get_user(user_id=dbc90e3cb55f431e91cd23da7b8958f2)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:08.097309 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dfad1b4-6220-4ef3-a588-5c9bfc5a6508 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:08.100909 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 67/131] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:19:07 2022] GET /identity/v3/users/dbc90e3cb55f431e91cd23da7b8958f2 => generated 275 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:08.101702 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfad1b4-6220-4ef3-a588-5c9bfc5a6508 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:08.157743 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:08.225673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dfad1b4-6220-4ef3-a588-5c9bfc5a6508 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:08.247625 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cce2eb1-1014-4a9b-97d8-51dee2d526e7 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:08.262170 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b377cebe-3efb-4949-9b87-308f0cdf95ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:08.263026 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b377cebe-3efb-4949-9b87-308f0cdf95ae None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:08.263315 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b377cebe-3efb-4949-9b87-308f0cdf95ae None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:08.263599 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b377cebe-3efb-4949-9b87-308f0cdf95ae None admin] PATH_INFO: `/v3/users/dbc90e3cb55f431e91cd23da7b8958f2` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:08.264802 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b377cebe-3efb-4949-9b87-308f0cdf95ae None admin] RBAC: Authorizing `identity:get_user(user_id=dbc90e3cb55f431e91cd23da7b8958f2)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:08.266387 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b377cebe-3efb-4949-9b87-308f0cdf95ae None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:08.268076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 68/132] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:19:08 2022] GET /identity/v3/users/dbc90e3cb55f431e91cd23da7b8958f2 => generated 275 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:08.269006 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b377cebe-3efb-4949-9b87-308f0cdf95ae None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:08.306837 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:08.344410 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d04dec7-50af-4d1c-8732-ff35500bea3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:08.345327 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d04dec7-50af-4d1c-8732-ff35500bea3c None admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:08.345620 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d04dec7-50af-4d1c-8732-ff35500bea3c None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:08.345908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d04dec7-50af-4d1c-8732-ff35500bea3c None admin] PATH_INFO: `/v3/projects/4857259d2a364dff960547dc0aac75e0/users/df85f597d0ba471381a5a8f9b6886705/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:08.352188 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d04dec7-50af-4d1c-8732-ff35500bea3c None admin] RBAC: Authorizing `identity:create_grant(project_id=4857259d2a364dff960547dc0aac75e0, user_id=df85f597d0ba471381a5a8f9b6886705, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:08.353662 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d04dec7-50af-4d1c-8732-ff35500bea3c None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:08.375826 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b377cebe-3efb-4949-9b87-308f0cdf95ae None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:08.411947 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c30835-d605-42e2-8f71-a9f724e32951 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:08.412870 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c30835-d605-42e2-8f71-a9f724e32951 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:08.416325 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c30835-d605-42e2-8f71-a9f724e32951 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:08.416325 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c30835-d605-42e2-8f71-a9f724e32951 None admin] PATH_INFO: `/v3/projects/4857259d2a364dff960547dc0aac75e0` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:08.416325 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c30835-d605-42e2-8f71-a9f724e32951 None admin] RBAC: Authorizing `identity:get_project(project_id=4857259d2a364dff960547dc0aac75e0)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:08.416325 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c30835-d605-42e2-8f71-a9f724e32951 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:08.418129 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 69/133] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:08 2022] GET /identity/v3/projects/4857259d2a364dff960547dc0aac75e0 => generated 298 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:08.418984 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c30835-d605-42e2-8f71-a9f724e32951 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:08.477622 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 65/134] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Jul 12 17:19:08 2022] PUT /identity/v3/projects/4857259d2a364dff960547dc0aac75e0/users/df85f597d0ba471381a5a8f9b6886705/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 454 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:19:08.478229 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d04dec7-50af-4d1c-8732-ff35500bea3c None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:08.587247 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c30835-d605-42e2-8f71-a9f724e32951 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:08.593367 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:08.640510 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-186f3d1a-4d60-4f9a-825a-ee45547b6454 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:08.641432 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186f3d1a-4d60-4f9a-825a-ee45547b6454 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:08.641934 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186f3d1a-4d60-4f9a-825a-ee45547b6454 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:08.642229 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186f3d1a-4d60-4f9a-825a-ee45547b6454 None admin] PATH_INFO: `/v3/projects/bdf9abb6d70a41acbad1d4060c21408f` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:08.644995 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-186f3d1a-4d60-4f9a-825a-ee45547b6454 None admin] RBAC: Authorizing `identity:get_project(project_id=bdf9abb6d70a41acbad1d4060c21408f)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:08.650780 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-186f3d1a-4d60-4f9a-825a-ee45547b6454 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:08.654003 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 70/135] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:08 2022] GET /identity/v3/projects/bdf9abb6d70a41acbad1d4060c21408f => generated 312 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:08.659432 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51c52b0-879d-4eaa-b2fe-34d19a16ed20 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:08.659735 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51c52b0-879d-4eaa-b2fe-34d19a16ed20 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:08.660026 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51c52b0-879d-4eaa-b2fe-34d19a16ed20 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:08.661936 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 71/136] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:08 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:08.667979 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795df352-a5d9-4c30-9eff-944faa0d89a6 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:08.668326 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795df352-a5d9-4c30-9eff-944faa0d89a6 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:08.668645 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795df352-a5d9-4c30-9eff-944faa0d89a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:08.690257 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-795df352-a5d9-4c30-9eff-944faa0d89a6 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:08.691504 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-795df352-a5d9-4c30-9eff-944faa0d89a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:08.708798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d04dec7-50af-4d1c-8732-ff35500bea3c None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:08.739019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:08.753248 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b726eda-2149-403b-ad6c-55a9227d7777 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:08.754148 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b726eda-2149-403b-ad6c-55a9227d7777 None admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:08.754469 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b726eda-2149-403b-ad6c-55a9227d7777 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:08.754818 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b726eda-2149-403b-ad6c-55a9227d7777 None admin] PATH_INFO: `/v3/projects/4857259d2a364dff960547dc0aac75e0/users/dbc90e3cb55f431e91cd23da7b8958f2/roles/0c239663092d459e8c9724963e78da76` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:08.758272 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b726eda-2149-403b-ad6c-55a9227d7777 None admin] RBAC: Authorizing `identity:create_grant(project_id=4857259d2a364dff960547dc0aac75e0, user_id=dbc90e3cb55f431e91cd23da7b8958f2, role_id=0c239663092d459e8c9724963e78da76)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:08.759825 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b726eda-2149-403b-ad6c-55a9227d7777 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:08.778721 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 72/137] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 12 17:19:08.779997 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-795df352-a5d9-4c30-9eff-944faa0d89a6 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:08.846954 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 66/138] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Jul 12 17:19:08 2022] PUT /identity/v3/projects/4857259d2a364dff960547dc0aac75e0/users/dbc90e3cb55f431e91cd23da7b8958f2/roles/0c239663092d459e8c9724963e78da76 => generated 0 bytes in 369 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:19:08.847695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b726eda-2149-403b-ad6c-55a9227d7777 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:08.915866 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-795df352-a5d9-4c30-9eff-944faa0d89a6 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:08.918270 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:08.970127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fd2978-20ab-46a3-a52c-896beb4bd786 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:08.973115 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fd2978-20ab-46a3-a52c-896beb4bd786 None admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:08.973420 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fd2978-20ab-46a3-a52c-896beb4bd786 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:08.973712 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fd2978-20ab-46a3-a52c-896beb4bd786 None admin] PATH_INFO: `/v3/projects/bdf9abb6d70a41acbad1d4060c21408f/users/dbc90e3cb55f431e91cd23da7b8958f2/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:08.977093 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0fd2978-20ab-46a3-a52c-896beb4bd786 None admin] RBAC: Authorizing `identity:create_grant(project_id=bdf9abb6d70a41acbad1d4060c21408f, user_id=dbc90e3cb55f431e91cd23da7b8958f2, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:08.979475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0fd2978-20ab-46a3-a52c-896beb4bd786 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:09.051920 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b726eda-2149-403b-ad6c-55a9227d7777 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:09.057697 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 73/139] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Jul 12 17:19:08 2022] PUT /identity/v3/projects/bdf9abb6d70a41acbad1d4060c21408f/users/dbc90e3cb55f431e91cd23da7b8958f2/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 278 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:19:09.108728 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61eff0f-2d84-466b-aebe-0df74637b72a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:09.109850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61eff0f-2d84-466b-aebe-0df74637b72a None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:09.110337 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61eff0f-2d84-466b-aebe-0df74637b72a None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:09.110822 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61eff0f-2d84-466b-aebe-0df74637b72a None admin] PATH_INFO: `/v3/domains/default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:09.115547 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61eff0f-2d84-466b-aebe-0df74637b72a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:09.117385 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61eff0f-2d84-466b-aebe-0df74637b72a None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:09.119484 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 67/140] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:19:08 2022] GET /identity/v3/domains/default => generated 199 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:09.125322 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fd2978-20ab-46a3-a52c-896beb4bd786 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:09.238163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1705c81-dd13-49a3-bbbd-acae1185affe None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:09.238770 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1705c81-dd13-49a3-bbbd-acae1185affe None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:09.239235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1705c81-dd13-49a3-bbbd-acae1185affe None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:09.248550 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 68/141] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:09 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 12 17:19:09.250394 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d9dd96-6a28-43a9-a2db-5dafe6b1c4e7 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:09.250953 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d9dd96-6a28-43a9-a2db-5dafe6b1c4e7 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:09.251449 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d9dd96-6a28-43a9-a2db-5dafe6b1c4e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:09.260557 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fd2978-20ab-46a3-a52c-896beb4bd786 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:09.288236 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-98d9dd96-6a28-43a9-a2db-5dafe6b1c4e7 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:09.289876 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-98d9dd96-6a28-43a9-a2db-5dafe6b1c4e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:09.314461 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03463fb5-1790-46ec-828a-946f2723a276 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:09.314461 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03463fb5-1790-46ec-828a-946f2723a276 None admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:09.314461 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03463fb5-1790-46ec-828a-946f2723a276 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:09.314461 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03463fb5-1790-46ec-828a-946f2723a276 None admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:09.314864 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03463fb5-1790-46ec-828a-946f2723a276 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:09.314864 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03463fb5-1790-46ec-828a-946f2723a276 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:09.386939 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:09.413363 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 69/142] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 12 17:19:09.437635 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d9dd96-6a28-43a9-a2db-5dafe6b1c4e7 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:09.478414 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 74/143] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Jul 12 17:19:09 2022] POST /identity/v3/users => generated 283 bytes in 354 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:19:09.547270 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d9dd96-6a28-43a9-a2db-5dafe6b1c4e7 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:09.598301 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7198be-a991-4f45-acfb-998623ccc9f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:09.599676 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7198be-a991-4f45-acfb-998623ccc9f1 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:09.599965 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7198be-a991-4f45-acfb-998623ccc9f1 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:09.600243 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7198be-a991-4f45-acfb-998623ccc9f1 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:09.624533 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7198be-a991-4f45-acfb-998623ccc9f1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:09.632636 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-2f7198be-a991-4f45-acfb-998623ccc9f1 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 12 17:19:09.632636 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:09.632636 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:09.632636 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:09.632636 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:09.632636 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:09.632636 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:09.632636 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:09.632636 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:09.632636 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:09.632636 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:09.632636 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:09.632636 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:09.632636 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:09.632636 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:09.632636 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:09.633648 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:09.633648 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:09.633648 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:09.633648 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:09.633648 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:09.633648 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:09.633648 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:09.633648 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:09.633648 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:09.633648 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:09.633648 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:09.633648 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:09.633648 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:09.633648 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:09.633648 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:09.633648 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:09.634865 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:09.634865 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:09.634865 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:09.634865 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:09.634865 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:09.634865 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:09.634865 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:09.634865 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:09.634865 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:09.634865 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:09.634865 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 12 17:19:09.634865 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:19:09.634865 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 70/144] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:09 2022] GET /identity/v3/roles/reader => generated 84 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:19:09.640145 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03463fb5-1790-46ec-828a-946f2723a276 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:09.691779 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:09.806578 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03463fb5-1790-46ec-828a-946f2723a276 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:09.846290 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c941099a-7121-491c-bcb0-bedcade3efdb None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:09.846822 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c941099a-7121-491c-bcb0-bedcade3efdb None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:09.846925 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c941099a-7121-491c-bcb0-bedcade3efdb None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:09.848258 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 71/145] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:09.852510 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f67170-1cc1-4ebc-8ace-7ea59784634b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:09.853202 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f67170-1cc1-4ebc-8ace-7ea59784634b None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:09.853492 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f67170-1cc1-4ebc-8ace-7ea59784634b None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:09.853806 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f67170-1cc1-4ebc-8ace-7ea59784634b None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:09.854605 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f67170-1cc1-4ebc-8ace-7ea59784634b None admin] RBAC: Adding query filter params (name=reader) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:09.854988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f67170-1cc1-4ebc-8ace-7ea59784634b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:09.855082 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b54233-02c8-4785-b41c-06502820e08b None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:09.855347 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b54233-02c8-4785-b41c-06502820e08b None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:09.855647 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b54233-02c8-4785-b41c-06502820e08b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:09.856387 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f67170-1cc1-4ebc-8ace-7ea59784634b None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:09.876569 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 75/146] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Jul 12 17:19:09 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:09.878574 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-e2b54233-02c8-4785-b41c-06502820e08b None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:09.879781 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-e2b54233-02c8-4785-b41c-06502820e08b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:09.883649 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f67170-1cc1-4ebc-8ace-7ea59784634b None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:09.941592 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:09.971032 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:09.992316 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f67170-1cc1-4ebc-8ace-7ea59784634b None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:10.007830 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 72/147] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 12 17:19:10.031516 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e271148c-ecc8-478e-8d3e-5e235b643c0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:10.031743 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b54233-02c8-4785-b41c-06502820e08b None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:10.032417 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e271148c-ecc8-478e-8d3e-5e235b643c0c None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:10.032730 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e271148c-ecc8-478e-8d3e-5e235b643c0c None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:10.033015 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e271148c-ecc8-478e-8d3e-5e235b643c0c None admin] PATH_INFO: `/v3/users/cf24bc418fc94f0f974e1870bd9bea31` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:10.034256 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e271148c-ecc8-478e-8d3e-5e235b643c0c None admin] RBAC: Authorizing `identity:get_user(user_id=cf24bc418fc94f0f974e1870bd9bea31)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:10.035801 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e271148c-ecc8-478e-8d3e-5e235b643c0c None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:10.038671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 76/148] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:19:09 2022] GET /identity/v3/users/cf24bc418fc94f0f974e1870bd9bea31 => generated 289 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:10.044999 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e271148c-ecc8-478e-8d3e-5e235b643c0c None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:10.155693 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e271148c-ecc8-478e-8d3e-5e235b643c0c None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:10.197448 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2341fb6f-f559-4db6-bc05-c6ba0a9de258 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:10.198372 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2341fb6f-f559-4db6-bc05-c6ba0a9de258 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:10.198711 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2341fb6f-f559-4db6-bc05-c6ba0a9de258 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:10.198998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2341fb6f-f559-4db6-bc05-c6ba0a9de258 None admin] PATH_INFO: `/v3/projects/4857259d2a364dff960547dc0aac75e0` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:10.200318 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2341fb6f-f559-4db6-bc05-c6ba0a9de258 None admin] RBAC: Authorizing `identity:get_project(project_id=4857259d2a364dff960547dc0aac75e0)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:10.201788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2341fb6f-f559-4db6-bc05-c6ba0a9de258 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:10.203588 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 77/149] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:10 2022] GET /identity/v3/projects/4857259d2a364dff960547dc0aac75e0 => generated 298 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:19:10.209413 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2341fb6f-f559-4db6-bc05-c6ba0a9de258 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:10.229030 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:10.271475 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b54233-02c8-4785-b41c-06502820e08b None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:10.319116 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2341fb6f-f559-4db6-bc05-c6ba0a9de258 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:10.357393 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3bad62d-27b0-4a34-b328-7a9912b00312 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:10.358586 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bad62d-27b0-4a34-b328-7a9912b00312 None admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:10.358970 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bad62d-27b0-4a34-b328-7a9912b00312 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:10.359262 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bad62d-27b0-4a34-b328-7a9912b00312 None admin] PATH_INFO: `/v3/projects/4857259d2a364dff960547dc0aac75e0/users/cf24bc418fc94f0f974e1870bd9bea31/roles/6670fba31f504efb9ae95813d9b6d660` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:10.362125 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3bad62d-27b0-4a34-b328-7a9912b00312 None admin] RBAC: Authorizing `identity:create_grant(project_id=4857259d2a364dff960547dc0aac75e0, user_id=cf24bc418fc94f0f974e1870bd9bea31, role_id=6670fba31f504efb9ae95813d9b6d660)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:10.363852 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3bad62d-27b0-4a34-b328-7a9912b00312 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:10.367154 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8610c7-a365-498d-a6fe-691641b85473 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:10.368005 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8610c7-a365-498d-a6fe-691641b85473 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:10.368307 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8610c7-a365-498d-a6fe-691641b85473 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:10.368577 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a8610c7-a365-498d-a6fe-691641b85473 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:10.386900 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a8610c7-a365-498d-a6fe-691641b85473 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:10.395318 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-5a8610c7-a365-498d-a6fe-691641b85473 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 12 17:19:10.395318 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:10.395318 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:10.395318 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:10.395318 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:10.395318 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:10.395318 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:10.395318 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:10.395318 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:10.395318 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:10.395318 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:10.395318 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:10.395318 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:10.395318 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:10.395318 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:10.395318 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:10.397153 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:10.397153 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:10.397153 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:10.397153 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:10.397153 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:10.397153 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:10.397153 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:10.397153 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:10.397153 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:10.397153 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:10.397153 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:10.397153 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:10.397153 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:10.397153 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:10.397153 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:10.397153 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:10.397908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:10.397908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:10.397908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:10.397908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:10.397908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:10.397908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:10.397908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:10.397908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:10.397908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:10.397908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:10.397908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 12 17:19:10.397908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:19:10.400179 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 73/150] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:10 2022] GET /identity/v3/roles/member => generated 84 bytes in 369 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:19:10.405899 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8610c7-a365-498d-a6fe-691641b85473 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:10.446772 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:10.469978 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 78/151] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Jul 12 17:19:10 2022] PUT /identity/v3/projects/4857259d2a364dff960547dc0aac75e0/users/cf24bc418fc94f0f974e1870bd9bea31/roles/6670fba31f504efb9ae95813d9b6d660 => generated 0 bytes in 261 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:19:10.530528 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a8610c7-a365-498d-a6fe-691641b85473 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:10.574064 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be6391dd-896f-40e6-a2ae-8facb1edd124 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:10.574797 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6391dd-896f-40e6-a2ae-8facb1edd124 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:10.575115 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6391dd-896f-40e6-a2ae-8facb1edd124 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:10.575379 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6391dd-896f-40e6-a2ae-8facb1edd124 None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:10.576171 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be6391dd-896f-40e6-a2ae-8facb1edd124 None admin] RBAC: Adding query filter params (name=member) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:10.576495 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be6391dd-896f-40e6-a2ae-8facb1edd124 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:10.577941 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be6391dd-896f-40e6-a2ae-8facb1edd124 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:10.594378 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 74/152] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Jul 12 17:19:10 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:10.600253 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3bad62d-27b0-4a34-b328-7a9912b00312 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:10.712810 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:10.744909 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3bad62d-27b0-4a34-b328-7a9912b00312 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:10.791128 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7147bcdd-0b68-41b6-a9c3-0d0543054c17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:10.792025 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7147bcdd-0b68-41b6-a9c3-0d0543054c17 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:10.792318 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7147bcdd-0b68-41b6-a9c3-0d0543054c17 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:10.792605 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7147bcdd-0b68-41b6-a9c3-0d0543054c17 None admin] PATH_INFO: `/v3/users/dbc90e3cb55f431e91cd23da7b8958f2` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:10.793840 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7147bcdd-0b68-41b6-a9c3-0d0543054c17 None admin] RBAC: Authorizing `identity:get_user(user_id=dbc90e3cb55f431e91cd23da7b8958f2)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:10.795396 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7147bcdd-0b68-41b6-a9c3-0d0543054c17 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:10.797067 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 79/153] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:19:10 2022] GET /identity/v3/users/dbc90e3cb55f431e91cd23da7b8958f2 => generated 275 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:10.803230 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be6391dd-896f-40e6-a2ae-8facb1edd124 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:10.929328 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:10.937903 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be6391dd-896f-40e6-a2ae-8facb1edd124 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:10.983714 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803a7b36-280f-4901-ad99-7b2228933d8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:10.984375 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803a7b36-280f-4901-ad99-7b2228933d8c None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:10.984659 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803a7b36-280f-4901-ad99-7b2228933d8c None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:10.984936 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803a7b36-280f-4901-ad99-7b2228933d8c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:11.074782 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-803a7b36-280f-4901-ad99-7b2228933d8c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:11.076170 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-803a7b36-280f-4901-ad99-7b2228933d8c None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:11.188235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:11.199573 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 75/154] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:19:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 398 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 12 17:19:11.205114 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7147bcdd-0b68-41b6-a9c3-0d0543054c17 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:11.345758 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7147bcdd-0b68-41b6-a9c3-0d0543054c17 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:11.392566 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba289741-07e1-421a-833e-6fcd3d4e7040 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:11.393903 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba289741-07e1-421a-833e-6fcd3d4e7040 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:11.394390 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba289741-07e1-421a-833e-6fcd3d4e7040 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:11.394906 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba289741-07e1-421a-833e-6fcd3d4e7040 None admin] PATH_INFO: `/v3/projects/4857259d2a364dff960547dc0aac75e0` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:11.396608 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba289741-07e1-421a-833e-6fcd3d4e7040 None admin] RBAC: Authorizing `identity:get_project(project_id=4857259d2a364dff960547dc0aac75e0)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:11.398269 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba289741-07e1-421a-833e-6fcd3d4e7040 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:11.400572 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 80/155] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:11 2022] GET /identity/v3/projects/4857259d2a364dff960547dc0aac75e0 => generated 298 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:11.406077 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803a7b36-280f-4901-ad99-7b2228933d8c None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:11.537355 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803a7b36-280f-4901-ad99-7b2228933d8c None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:11.577929 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29ecdc7-91e8-49fb-b05b-49116514949a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:11.578634 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29ecdc7-91e8-49fb-b05b-49116514949a None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:11.578929 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29ecdc7-91e8-49fb-b05b-49116514949a None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:11.579207 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29ecdc7-91e8-49fb-b05b-49116514949a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:11.580065 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29ecdc7-91e8-49fb-b05b-49116514949a None admin] RBAC: Adding query filter params (role.id=4c5d52f873f94dc1b10918068376add5, scope.project.id=4857259d2a364dff960547dc0aac75e0, user.id=dbc90e3cb55f431e91cd23da7b8958f2) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:11.580385 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29ecdc7-91e8-49fb-b05b-49116514949a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:11.581852 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a29ecdc7-91e8-49fb-b05b-49116514949a None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:11.589243 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 76/156] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:19:11 2022] GET /identity/v3/role_assignments?user.id=dbc90e3cb55f431e91cd23da7b8958f2&scope.project.id=4857259d2a364dff960547dc0aac75e0&role.id=4c5d52f873f94dc1b10918068376add5 => generated 608 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:11.798812 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931023bd-c693-41a1-a959-0f2122d236d1 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:11.799855 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931023bd-c693-41a1-a959-0f2122d236d1 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:11.800348 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931023bd-c693-41a1-a959-0f2122d236d1 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:11.802658 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 81/157] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:11 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:11.808295 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea20af21-91d8-4ce5-974a-4273eb67d1ea None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:11.808593 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea20af21-91d8-4ce5-974a-4273eb67d1ea None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:11.808882 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea20af21-91d8-4ce5-974a-4273eb67d1ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:11.830956 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-ea20af21-91d8-4ce5-974a-4273eb67d1ea None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:11.832166 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-ea20af21-91d8-4ce5-974a-4273eb67d1ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:11.906038 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:11.929503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 77/158] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:11.978751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-931023bd-c693-41a1-a959-0f2122d236d1 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:12.002969 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:12.121937 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-931023bd-c693-41a1-a959-0f2122d236d1 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:12.165410 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f5354b-59c9-4ea9-88c4-0ed46469381e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:12.166222 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f5354b-59c9-4ea9-88c4-0ed46469381e None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:12.166533 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f5354b-59c9-4ea9-88c4-0ed46469381e None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:12.166815 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f5354b-59c9-4ea9-88c4-0ed46469381e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:12.180682 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5f5354b-59c9-4ea9-88c4-0ed46469381e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:12.188738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-b5f5354b-59c9-4ea9-88c4-0ed46469381e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:19:12.188738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:12.188738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:12.188738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:12.188738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:12.188738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:12.188738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:12.188738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:12.188738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:12.188738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:12.188738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:12.188738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:12.188738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:12.188738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:12.188738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:12.188738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:12.189619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:12.189619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:12.189619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:12.189619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:12.189619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:12.189619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:12.189619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:12.189619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:12.189619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:12.189619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:12.189619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:12.189619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:12.189619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:12.189619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:12.189619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:12.189619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:12.190523 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:12.190523 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:12.190523 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:12.190523 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:12.190523 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:12.190523 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:12.190523 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:12.190523 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:12.190523 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:12.190523 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:12.190523 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:19:12.190523 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:12.190523 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 82/159] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Jul 12 17:19:11 2022] GET /identity/v3/roles/admin => generated 83 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:12.196381 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea20af21-91d8-4ce5-974a-4273eb67d1ea None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:12.323832 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea20af21-91d8-4ce5-974a-4273eb67d1ea None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:12.365706 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea08bad3-a889-4b9b-9a0b-812fbee51e8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:12.366469 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea08bad3-a889-4b9b-9a0b-812fbee51e8a None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:12.366765 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea08bad3-a889-4b9b-9a0b-812fbee51e8a None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:12.367093 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea08bad3-a889-4b9b-9a0b-812fbee51e8a None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:12.367863 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea08bad3-a889-4b9b-9a0b-812fbee51e8a None admin] RBAC: Adding query filter params (name=admin) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:12.368176 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea08bad3-a889-4b9b-9a0b-812fbee51e8a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:12.369672 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea08bad3-a889-4b9b-9a0b-812fbee51e8a None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:12.383557 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 78/160] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:12 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:12.389088 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f5354b-59c9-4ea9-88c4-0ed46469381e None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:12.473241 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:12.516108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f5354b-59c9-4ea9-88c4-0ed46469381e None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:12.568817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c226e219-0d35-4913-ba1b-e64728f376f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:12.569719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c226e219-0d35-4913-ba1b-e64728f376f9 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:12.570014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c226e219-0d35-4913-ba1b-e64728f376f9 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:12.570298 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c226e219-0d35-4913-ba1b-e64728f376f9 None admin] PATH_INFO: `/v3/users/df85f597d0ba471381a5a8f9b6886705` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:12.573719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c226e219-0d35-4913-ba1b-e64728f376f9 None admin] RBAC: Authorizing `identity:get_user(user_id=df85f597d0ba471381a5a8f9b6886705)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:12.575317 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c226e219-0d35-4913-ba1b-e64728f376f9 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:12.577386 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 83/161] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:19:12 2022] GET /identity/v3/users/df85f597d0ba471381a5a8f9b6886705 => generated 247 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:12.588246 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea08bad3-a889-4b9b-9a0b-812fbee51e8a None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:12.664836 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:12.739501 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea08bad3-a889-4b9b-9a0b-812fbee51e8a None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:12.765072 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fdb5a73-b2ba-4fa4-b899-b84390ed7485 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:12.765364 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fdb5a73-b2ba-4fa4-b899-b84390ed7485 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:12.765643 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fdb5a73-b2ba-4fa4-b899-b84390ed7485 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:12.767206 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 84/162] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:12.773048 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e00504-b7ae-46ac-92cd-6c51356b6bea None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:12.773345 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e00504-b7ae-46ac-92cd-6c51356b6bea None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:12.773633 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e00504-b7ae-46ac-92cd-6c51356b6bea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:12.785113 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01c9714-3e91-4f64-92cc-18e8177b18b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:12.785777 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01c9714-3e91-4f64-92cc-18e8177b18b0 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:12.786105 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01c9714-3e91-4f64-92cc-18e8177b18b0 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:12.787149 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01c9714-3e91-4f64-92cc-18e8177b18b0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:12.805156 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-b2e00504-b7ae-46ac-92cd-6c51356b6bea None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:12.806504 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-b2e00504-b7ae-46ac-92cd-6c51356b6bea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:12.877850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01c9714-3e91-4f64-92cc-18e8177b18b0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:12.879696 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01c9714-3e91-4f64-92cc-18e8177b18b0 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:12.937383 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:12.946106 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 85/163] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:12.986268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e00504-b7ae-46ac-92cd-6c51356b6bea None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:13.015038 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 79/164] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:19:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 427 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 12 17:19:13.021589 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01c9714-3e91-4f64-92cc-18e8177b18b0 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:13.065509 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:13.101580 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e00504-b7ae-46ac-92cd-6c51356b6bea None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:13.146210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eedb314b-b5d5-464f-ad9b-d11aa264eb91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:13.147068 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedb314b-b5d5-464f-ad9b-d11aa264eb91 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:13.148240 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedb314b-b5d5-464f-ad9b-d11aa264eb91 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:13.148539 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedb314b-b5d5-464f-ad9b-d11aa264eb91 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:13.173089 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eedb314b-b5d5-464f-ad9b-d11aa264eb91 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:13.183014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-eedb314b-b5d5-464f-ad9b-d11aa264eb91 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 12 17:19:13.183014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:13.183014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:13.183014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:13.183014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:13.183014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:13.183014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:13.183014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:13.183014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:13.183014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:13.183014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:13.183014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:13.183014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:13.183014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:13.183014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:13.183014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:13.183969 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:13.183969 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:13.183969 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:13.183969 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:13.183969 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:13.183969 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:13.183969 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:13.183969 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:13.183969 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:13.183969 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:13.183969 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:13.183969 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:13.183969 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:13.183969 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:13.183969 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:13.183969 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:13.184751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:13.184751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:13.184751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:13.184751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:13.184751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:13.184751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:13.184751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:13.184751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:13.184751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:13.184751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:13.184751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 12 17:19:13.184751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:13.184751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 86/165] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Jul 12 17:19:12 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:13.186190 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f1b230-f70a-43e3-965f-302d0ef3a1f8 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:13.186190 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f1b230-f70a-43e3-965f-302d0ef3a1f8 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:13.188163 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f1b230-f70a-43e3-965f-302d0ef3a1f8 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:13.188163 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 87/166] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:13.193563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f1b230-f70a-43e3-965f-302d0ef3a1f8 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:13.349600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01c9714-3e91-4f64-92cc-18e8177b18b0 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:13.424157 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f816894-b6f2-4cd0-8ac0-5b1c9c5dbdc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:13.425037 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f816894-b6f2-4cd0-8ac0-5b1c9c5dbdc5 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:13.425333 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f816894-b6f2-4cd0-8ac0-5b1c9c5dbdc5 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:13.425613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f816894-b6f2-4cd0-8ac0-5b1c9c5dbdc5 None admin] PATH_INFO: `/v3/projects/4857259d2a364dff960547dc0aac75e0` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:13.430792 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f816894-b6f2-4cd0-8ac0-5b1c9c5dbdc5 None admin] RBAC: Authorizing `identity:get_project(project_id=4857259d2a364dff960547dc0aac75e0)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:13.432284 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f816894-b6f2-4cd0-8ac0-5b1c9c5dbdc5 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:13.434230 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 80/167] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:13 2022] GET /identity/v3/projects/4857259d2a364dff960547dc0aac75e0 => generated 298 bytes in 414 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:19:13.435366 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c33241-fcb0-45b5-98d1-d3a0dd69bf12 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:13.435734 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c33241-fcb0-45b5-98d1-d3a0dd69bf12 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:13.436171 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c33241-fcb0-45b5-98d1-d3a0dd69bf12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:13.466008 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-00c33241-fcb0-45b5-98d1-d3a0dd69bf12 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:13.467420 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-00c33241-fcb0-45b5-98d1-d3a0dd69bf12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:13.477336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f1b230-f70a-43e3-965f-302d0ef3a1f8 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:13.479114 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:13.572675 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 81/168] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:13.573582 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c33241-fcb0-45b5-98d1-d3a0dd69bf12 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:13.577653 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:13.593039 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7177034-f7d2-44f4-a307-330978421c3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:13.593731 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7177034-f7d2-44f4-a307-330978421c3b None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:13.594023 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7177034-f7d2-44f4-a307-330978421c3b None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:13.594337 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7177034-f7d2-44f4-a307-330978421c3b None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:13.595145 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7177034-f7d2-44f4-a307-330978421c3b None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:13.595462 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7177034-f7d2-44f4-a307-330978421c3b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:13.597033 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7177034-f7d2-44f4-a307-330978421c3b None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:13.607596 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 88/169] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:19:13 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 414 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:13.613763 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7177034-f7d2-44f4-a307-330978421c3b None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:13.784508 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7177034-f7d2-44f4-a307-330978421c3b None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:13.800427 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c33241-fcb0-45b5-98d1-d3a0dd69bf12 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:13.861372 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69bbeb7-84cc-4f3b-8fea-f3236acbb020 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:13.862038 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69bbeb7-84cc-4f3b-8fea-f3236acbb020 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:13.862324 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69bbeb7-84cc-4f3b-8fea-f3236acbb020 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:13.862681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69bbeb7-84cc-4f3b-8fea-f3236acbb020 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:13.863634 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69bbeb7-84cc-4f3b-8fea-f3236acbb020 None admin] RBAC: Adding query filter params (role.id=eb05bf286c894ea9bf44a6c023a4323b, scope.project.id=4857259d2a364dff960547dc0aac75e0, user.id=df85f597d0ba471381a5a8f9b6886705) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:13.863956 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69bbeb7-84cc-4f3b-8fea-f3236acbb020 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:13.865654 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69bbeb7-84cc-4f3b-8fea-f3236acbb020 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:13.878301 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 82/170] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:19:13 2022] GET /identity/v3/role_assignments?user.id=df85f597d0ba471381a5a8f9b6886705&scope.project.id=4857259d2a364dff960547dc0aac75e0&role.id=eb05bf286c894ea9bf44a6c023a4323b => generated 608 bytes in 305 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:13.878971 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69bbeb7-84cc-4f3b-8fea-f3236acbb020 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:13.880856 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-571ec1a1-b113-4ba6-a3bf-c4d9def6bd62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:13.882576 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571ec1a1-b113-4ba6-a3bf-c4d9def6bd62 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:13.883092 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571ec1a1-b113-4ba6-a3bf-c4d9def6bd62 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:13.883567 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571ec1a1-b113-4ba6-a3bf-c4d9def6bd62 None admin] PATH_INFO: `/v3/users/dbc90e3cb55f431e91cd23da7b8958f2` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:13.891416 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-571ec1a1-b113-4ba6-a3bf-c4d9def6bd62 None admin] RBAC: Authorizing `identity:get_user(user_id=dbc90e3cb55f431e91cd23da7b8958f2)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:13.893284 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-571ec1a1-b113-4ba6-a3bf-c4d9def6bd62 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:13.895359 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 89/171] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:19:13 2022] GET /identity/v3/users/dbc90e3cb55f431e91cd23da7b8958f2 => generated 275 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:13.901860 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-571ec1a1-b113-4ba6-a3bf-c4d9def6bd62 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:13.910764 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:14.033350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69bbeb7-84cc-4f3b-8fea-f3236acbb020 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:14.074216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33acc6a6-b05d-4675-b288-44fdc9532276 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:14.075160 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33acc6a6-b05d-4675-b288-44fdc9532276 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:14.075597 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33acc6a6-b05d-4675-b288-44fdc9532276 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:14.075976 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33acc6a6-b05d-4675-b288-44fdc9532276 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:14.078372 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:14.089281 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33acc6a6-b05d-4675-b288-44fdc9532276 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:14.097908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-33acc6a6-b05d-4675-b288-44fdc9532276 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 12 17:19:14.097908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:14.097908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:14.097908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:14.097908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:14.097908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:14.097908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:14.097908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:14.097908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:14.097908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:14.097908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:14.097908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:14.097908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:14.097908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:14.097908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:14.097908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:14.099158 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:14.099158 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:14.099158 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:14.099158 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:14.099158 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:14.099158 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:14.099158 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:14.099158 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:14.099158 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:14.099158 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:14.099158 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:14.099158 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:14.099158 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:14.099158 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:14.099158 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:14.099158 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:14.100205 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:14.100205 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:14.100205 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:14.100205 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:14.100205 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:14.100205 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:14.100205 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:14.100205 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:14.100205 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:14.100205 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:14.100205 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 12 17:19:14.100205 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:19:14.102208 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 83/172] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:13 2022] GET /identity/v3/roles/member => generated 84 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:14.108104 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f234d523-2199-4e52-8976-ff7ef03e6066 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:14.108538 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f234d523-2199-4e52-8976-ff7ef03e6066 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:14.108924 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f234d523-2199-4e52-8976-ff7ef03e6066 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:14.114484 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 84/173] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:14 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:14.114484 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5b300e-0f40-4bdd-a090-237576921ba7 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:14.114484 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5b300e-0f40-4bdd-a090-237576921ba7 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:14.114484 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5b300e-0f40-4bdd-a090-237576921ba7 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:14.114818 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 85/174] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:14.115818 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b5b300e-0f40-4bdd-a090-237576921ba7 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:14.147169 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-571ec1a1-b113-4ba6-a3bf-c4d9def6bd62 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:14.185409 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9d4e42-6c50-46c5-a5b3-c628ae6b47e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:14.186400 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9d4e42-6c50-46c5-a5b3-c628ae6b47e1 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:14.186886 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9d4e42-6c50-46c5-a5b3-c628ae6b47e1 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:14.187363 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9d4e42-6c50-46c5-a5b3-c628ae6b47e1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:14.220267 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:14.267422 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b5b300e-0f40-4bdd-a090-237576921ba7 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:14.280641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9d4e42-6c50-46c5-a5b3-c628ae6b47e1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:14.282210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9d4e42-6c50-46c5-a5b3-c628ae6b47e1 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:14.314231 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa6c3d4-ec71-4db4-84a0-bc7641bedcd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:14.315526 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa6c3d4-ec71-4db4-84a0-bc7641bedcd8 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:14.316016 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa6c3d4-ec71-4db4-84a0-bc7641bedcd8 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:14.316618 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa6c3d4-ec71-4db4-84a0-bc7641bedcd8 None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:14.318589 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa6c3d4-ec71-4db4-84a0-bc7641bedcd8 None admin] RBAC: Adding query filter params (name=member) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:14.319387 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa6c3d4-ec71-4db4-84a0-bc7641bedcd8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:14.321158 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa6c3d4-ec71-4db4-84a0-bc7641bedcd8 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:14.333377 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 86/175] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Jul 12 17:19:14 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:14.335756 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070e0853-eb6b-4b18-9fff-b08c665104c5 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:14.336248 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070e0853-eb6b-4b18-9fff-b08c665104c5 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:14.336733 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070e0853-eb6b-4b18-9fff-b08c665104c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:14.359619 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-070e0853-eb6b-4b18-9fff-b08c665104c5 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:14.360122 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-070e0853-eb6b-4b18-9fff-b08c665104c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:14.363326 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:14.424959 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 90/176] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:19:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 524 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 12 17:19:14.427816 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d78839f-dea7-4cfe-84b6-d744e978cd06 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:14.428118 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d78839f-dea7-4cfe-84b6-d744e978cd06 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:14.428406 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d78839f-dea7-4cfe-84b6-d744e978cd06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:14.439945 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 87/177] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:14.441805 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070e0853-eb6b-4b18-9fff-b08c665104c5 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:14.463504 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-1d78839f-dea7-4cfe-84b6-d744e978cd06 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:14.464730 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-1d78839f-dea7-4cfe-84b6-d744e978cd06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:14.526850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:14.547126 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 91/178] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:14.547817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d78839f-dea7-4cfe-84b6-d744e978cd06 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:14.584425 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-070e0853-eb6b-4b18-9fff-b08c665104c5 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:14.629927 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08713d01-fc55-467b-b2fc-721e770bed3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:14.629927 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08713d01-fc55-467b-b2fc-721e770bed3c None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:14.629927 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08713d01-fc55-467b-b2fc-721e770bed3c None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:14.629927 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08713d01-fc55-467b-b2fc-721e770bed3c None admin] PATH_INFO: `/v3/users/dbc90e3cb55f431e91cd23da7b8958f2` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:14.631751 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08713d01-fc55-467b-b2fc-721e770bed3c None admin] RBAC: Authorizing `identity:get_user(user_id=dbc90e3cb55f431e91cd23da7b8958f2)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:14.633965 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08713d01-fc55-467b-b2fc-721e770bed3c None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:14.640072 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 88/179] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:19:14 2022] GET /identity/v3/users/dbc90e3cb55f431e91cd23da7b8958f2 => generated 275 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:14.640072 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08713d01-fc55-467b-b2fc-721e770bed3c None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:14.654656 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:14.665016 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d78839f-dea7-4cfe-84b6-d744e978cd06 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:14.700433 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c20a37-4b86-44c4-a686-5d890dbb4896 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:14.701534 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c20a37-4b86-44c4-a686-5d890dbb4896 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:14.701980 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c20a37-4b86-44c4-a686-5d890dbb4896 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:14.702260 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c20a37-4b86-44c4-a686-5d890dbb4896 None admin] PATH_INFO: `/v3/projects/4857259d2a364dff960547dc0aac75e0` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:14.703692 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c20a37-4b86-44c4-a686-5d890dbb4896 None admin] RBAC: Authorizing `identity:get_project(project_id=4857259d2a364dff960547dc0aac75e0)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:14.705945 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c20a37-4b86-44c4-a686-5d890dbb4896 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:14.709078 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 92/180] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:14 2022] GET /identity/v3/projects/4857259d2a364dff960547dc0aac75e0 => generated 298 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:14.709934 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c20a37-4b86-44c4-a686-5d890dbb4896 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:14.748918 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08713d01-fc55-467b-b2fc-721e770bed3c None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:14.778037 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:14.783679 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c3f30c-9d01-49bd-ba94-db47bec2dfde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:14.784476 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c3f30c-9d01-49bd-ba94-db47bec2dfde None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:14.788496 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c3f30c-9d01-49bd-ba94-db47bec2dfde None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:14.788496 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c3f30c-9d01-49bd-ba94-db47bec2dfde None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:14.795162 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c3f30c-9d01-49bd-ba94-db47bec2dfde None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:14.802863 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-40c3f30c-9d01-49bd-ba94-db47bec2dfde None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:19:14.802863 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:14.802863 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:14.802863 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:14.802863 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:14.802863 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:14.802863 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:14.802863 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:14.802863 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:14.802863 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:14.802863 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:14.802863 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:14.802863 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:14.802863 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:14.802863 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:14.802863 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:14.803950 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:14.803950 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:14.803950 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:14.803950 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:14.803950 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:14.803950 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:14.803950 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:14.803950 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:14.803950 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:14.803950 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:14.803950 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:14.803950 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:14.803950 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:14.803950 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:14.803950 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:14.803950 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:14.804883 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:14.804883 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:14.804883 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:14.804883 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:14.804883 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:14.804883 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:14.804883 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:14.804883 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:14.804883 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:14.804883 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:14.804883 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:19:14.804883 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:19:14.804883 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 89/181] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Jul 12 17:19:14 2022] GET /identity/v3/roles/admin => generated 83 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:14.805710 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25200b3-63dd-4a55-9af3-397e77a31d0a None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:14.805777 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25200b3-63dd-4a55-9af3-397e77a31d0a None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:14.806011 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25200b3-63dd-4a55-9af3-397e77a31d0a None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:14.807302 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 90/182] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:14 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:14.811891 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25200b3-63dd-4a55-9af3-397e77a31d0a None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:14.819166 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c20a37-4b86-44c4-a686-5d890dbb4896 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:14.854667 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34331866-99d4-40fe-b4c8-a77ec528b8ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:14.855382 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34331866-99d4-40fe-b4c8-a77ec528b8ad None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:14.855668 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34331866-99d4-40fe-b4c8-a77ec528b8ad None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:14.855952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34331866-99d4-40fe-b4c8-a77ec528b8ad None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:14.866713 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34331866-99d4-40fe-b4c8-a77ec528b8ad None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:14.874393 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-34331866-99d4-40fe-b4c8-a77ec528b8ad None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 12 17:19:14.874393 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:14.874393 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:14.874393 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:14.874393 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:14.874393 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:14.874393 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:14.874393 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:14.874393 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:14.874393 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:14.874393 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:14.874393 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:14.874393 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:14.874393 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:14.874393 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:14.874393 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:14.875336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:14.875336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:14.875336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:14.875336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:14.875336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:14.875336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:14.875336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:14.875336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:14.875336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:14.875336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:14.875336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:14.875336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:14.875336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:14.875336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:14.875336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:14.875336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:14.876127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:14.876127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:14.876127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:14.876127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:14.876127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:14.876127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:14.876127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:14.876127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:14.876127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:14.876127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:14.876127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 12 17:19:14.876127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:14.876127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 93/183] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:14 2022] GET /identity/v3/roles/reader => generated 84 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:14.876849 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34331866-99d4-40fe-b4c8-a77ec528b8ad None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:14.900100 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:14.917014 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25200b3-63dd-4a55-9af3-397e77a31d0a None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:14.951199 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6628a8f-d721-4c66-bde8-746ae2a5e73e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:14.951947 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6628a8f-d721-4c66-bde8-746ae2a5e73e None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:14.952240 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6628a8f-d721-4c66-bde8-746ae2a5e73e None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:14.952528 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6628a8f-d721-4c66-bde8-746ae2a5e73e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:14.980424 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34331866-99d4-40fe-b4c8-a77ec528b8ad None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:15.061428 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:15.067814 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04af6ef9-6af7-4170-bb00-c276d03ad313 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:15.068740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04af6ef9-6af7-4170-bb00-c276d03ad313 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:15.068858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04af6ef9-6af7-4170-bb00-c276d03ad313 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:15.069104 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04af6ef9-6af7-4170-bb00-c276d03ad313 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:15.073868 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04af6ef9-6af7-4170-bb00-c276d03ad313 None admin] RBAC: Adding query filter params (role.id=0c239663092d459e8c9724963e78da76, scope.project.id=4857259d2a364dff960547dc0aac75e0, user.id=dbc90e3cb55f431e91cd23da7b8958f2) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:15.073868 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04af6ef9-6af7-4170-bb00-c276d03ad313 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:15.073868 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04af6ef9-6af7-4170-bb00-c276d03ad313 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:15.074086 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6628a8f-d721-4c66-bde8-746ae2a5e73e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:15.074086 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6628a8f-d721-4c66-bde8-746ae2a5e73e None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:15.077633 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 94/184] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:19:14 2022] GET /identity/v3/role_assignments?user.id=dbc90e3cb55f431e91cd23da7b8958f2&scope.project.id=4857259d2a364dff960547dc0aac75e0&role.id=0c239663092d459e8c9724963e78da76 => generated 608 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:15.087820 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07242003-6d7f-4dc7-994b-79dbcb069378 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:15.087820 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07242003-6d7f-4dc7-994b-79dbcb069378 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:15.087820 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07242003-6d7f-4dc7-994b-79dbcb069378 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:15.087820 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 95/185] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:15 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:15.087820 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07242003-6d7f-4dc7-994b-79dbcb069378 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:15.189258 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:15.198701 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 91/186] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:19:14 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 12 17:19:15.201316 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e78dde8-3d57-4824-805f-660c5ee07855 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:15.201611 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e78dde8-3d57-4824-805f-660c5ee07855 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:15.201894 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e78dde8-3d57-4824-805f-660c5ee07855 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:15.212755 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07242003-6d7f-4dc7-994b-79dbcb069378 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:15.221008 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-0e78dde8-3d57-4824-805f-660c5ee07855 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:15.221803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-0e78dde8-3d57-4824-805f-660c5ee07855 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:15.260873 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292f8975-782b-4c6c-a6e5-e4be7d943a08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:15.261247 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292f8975-782b-4c6c-a6e5-e4be7d943a08 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:15.261535 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292f8975-782b-4c6c-a6e5-e4be7d943a08 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:15.261817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292f8975-782b-4c6c-a6e5-e4be7d943a08 None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:15.262704 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292f8975-782b-4c6c-a6e5-e4be7d943a08 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:15.263130 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292f8975-782b-4c6c-a6e5-e4be7d943a08 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:15.264539 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292f8975-782b-4c6c-a6e5-e4be7d943a08 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:15.274783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 96/187] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:15 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:15.276867 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292f8975-782b-4c6c-a6e5-e4be7d943a08 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:15.307520 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 92/188] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:15.309032 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a3f9a0-2c21-42e4-8821-9274fd883e2c None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:15.309331 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a3f9a0-2c21-42e4-8821-9274fd883e2c None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:15.309615 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a3f9a0-2c21-42e4-8821-9274fd883e2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:15.331236 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-49a3f9a0-2c21-42e4-8821-9274fd883e2c None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:15.332698 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-49a3f9a0-2c21-42e4-8821-9274fd883e2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:15.347493 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:15.390710 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292f8975-782b-4c6c-a6e5-e4be7d943a08 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:15.400663 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 93/189] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:15.407848 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a3f9a0-2c21-42e4-8821-9274fd883e2c None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:15.434142 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0594dc-fba2-4575-ad55-c773dcdc16ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:15.434850 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0594dc-fba2-4575-ad55-c773dcdc16ba None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:15.435139 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0594dc-fba2-4575-ad55-c773dcdc16ba None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:15.435416 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0594dc-fba2-4575-ad55-c773dcdc16ba None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:15.436172 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce0594dc-fba2-4575-ad55-c773dcdc16ba None admin] RBAC: Adding query filter params (name=reader) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:15.436484 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce0594dc-fba2-4575-ad55-c773dcdc16ba None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:15.437855 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce0594dc-fba2-4575-ad55-c773dcdc16ba None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:15.447228 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 97/190] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Jul 12 17:19:15 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:15.447725 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0594dc-fba2-4575-ad55-c773dcdc16ba None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:15.456627 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:15.506870 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a3f9a0-2c21-42e4-8821-9274fd883e2c None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:15.540922 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bf8f75-8b06-4c0f-a03a-59285056d66f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:15.541787 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bf8f75-8b06-4c0f-a03a-59285056d66f None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:15.542096 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bf8f75-8b06-4c0f-a03a-59285056d66f None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:15.542376 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bf8f75-8b06-4c0f-a03a-59285056d66f None admin] PATH_INFO: `/v3/projects/bdf9abb6d70a41acbad1d4060c21408f` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:15.543601 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7bf8f75-8b06-4c0f-a03a-59285056d66f None admin] RBAC: Authorizing `identity:get_project(project_id=bdf9abb6d70a41acbad1d4060c21408f)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:15.545062 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7bf8f75-8b06-4c0f-a03a-59285056d66f None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:15.546796 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 94/191] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:15 2022] GET /identity/v3/projects/bdf9abb6d70a41acbad1d4060c21408f => generated 312 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:15.549564 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0594dc-fba2-4575-ad55-c773dcdc16ba None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:15.551502 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bf8f75-8b06-4c0f-a03a-59285056d66f None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:15.571693 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:15.585613 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07107a0-2cc0-4ec9-b63f-1cb12ffb9eaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:15.586538 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07107a0-2cc0-4ec9-b63f-1cb12ffb9eaf None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:15.588690 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07107a0-2cc0-4ec9-b63f-1cb12ffb9eaf None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:15.588690 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07107a0-2cc0-4ec9-b63f-1cb12ffb9eaf None admin] PATH_INFO: `/v3/users/84726da6e6d246e5b9500096bf044a36` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:15.602570 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07107a0-2cc0-4ec9-b63f-1cb12ffb9eaf None admin] RBAC: Authorizing `identity:get_user(user_id=84726da6e6d246e5b9500096bf044a36)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:15.604031 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07107a0-2cc0-4ec9-b63f-1cb12ffb9eaf None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:15.605965 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 98/192] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:19:15 2022] GET /identity/v3/users/84726da6e6d246e5b9500096bf044a36 => generated 283 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:15.606560 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07107a0-2cc0-4ec9-b63f-1cb12ffb9eaf None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:15.656687 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bf8f75-8b06-4c0f-a03a-59285056d66f None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:15.673268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:15.691217 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76a135b-504b-43ce-9f35-0448ca5d1b06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:15.692021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76a135b-504b-43ce-9f35-0448ca5d1b06 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:15.692888 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76a135b-504b-43ce-9f35-0448ca5d1b06 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:15.693168 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76a135b-504b-43ce-9f35-0448ca5d1b06 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:15.703270 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d76a135b-504b-43ce-9f35-0448ca5d1b06 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:15.711397 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-d76a135b-504b-43ce-9f35-0448ca5d1b06 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 12 17:19:15.711397 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:15.711397 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:15.711397 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:15.711397 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:15.711397 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:15.711397 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:15.711397 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:15.711397 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:15.711397 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:15.711397 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:15.711397 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:15.711397 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:15.711397 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:15.711397 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:15.711397 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:15.712315 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:15.712315 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:15.712315 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:15.712315 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:15.712315 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:15.712315 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:15.712315 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:15.712315 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:15.712315 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:15.712315 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:15.712315 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:15.712315 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:15.712315 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:15.712315 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:15.712315 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:15.712315 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:15.713124 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:15.713124 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:15.713124 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:15.713124 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:15.713124 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:15.713124 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:15.713124 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:15.713124 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:15.713124 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:15.713124 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:15.713124 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 12 17:19:15.713124 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:19:15.713761 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07107a0-2cc0-4ec9-b63f-1cb12ffb9eaf None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:15.713842 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 95/193] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Jul 12 17:19:15 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:15.713842 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76a135b-504b-43ce-9f35-0448ca5d1b06 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:15.751433 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be2279b-d362-4608-b5f7-10de838b448b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:15.752235 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be2279b-d362-4608-b5f7-10de838b448b None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:15.752521 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be2279b-d362-4608-b5f7-10de838b448b None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:15.752798 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be2279b-d362-4608-b5f7-10de838b448b None admin] PATH_INFO: `/v3/domains/default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:15.753965 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3be2279b-d362-4608-b5f7-10de838b448b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:15.756080 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3be2279b-d362-4608-b5f7-10de838b448b None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:15.757858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 99/194] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:19:15 2022] GET /identity/v3/domains/default => generated 199 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:15.758481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be2279b-d362-4608-b5f7-10de838b448b None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:15.795177 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:15.816306 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76a135b-504b-43ce-9f35-0448ca5d1b06 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:15.850606 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a371943d-39e6-4375-8763-e950ad38d15e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:15.851520 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a371943d-39e6-4375-8763-e950ad38d15e None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:15.851804 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a371943d-39e6-4375-8763-e950ad38d15e None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:15.852082 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a371943d-39e6-4375-8763-e950ad38d15e None admin] PATH_INFO: `/v3/users/cf24bc418fc94f0f974e1870bd9bea31` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:15.853268 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a371943d-39e6-4375-8763-e950ad38d15e None admin] RBAC: Authorizing `identity:get_user(user_id=cf24bc418fc94f0f974e1870bd9bea31)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:15.854733 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a371943d-39e6-4375-8763-e950ad38d15e None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:15.856346 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 96/195] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:19:15 2022] GET /identity/v3/users/cf24bc418fc94f0f974e1870bd9bea31 => generated 289 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:15.857097 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a371943d-39e6-4375-8763-e950ad38d15e None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:15.861855 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be2279b-d362-4608-b5f7-10de838b448b None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:15.891568 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:15.896532 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a03385e-45e0-4b36-af05-a8cf5395d6c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:15.897301 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a03385e-45e0-4b36-af05-a8cf5395d6c2 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:15.897584 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a03385e-45e0-4b36-af05-a8cf5395d6c2 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:15.898630 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a03385e-45e0-4b36-af05-a8cf5395d6c2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:15.898766 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a03385e-45e0-4b36-af05-a8cf5395d6c2 None admin] RBAC: Adding query filter params (role.id=4c5d52f873f94dc1b10918068376add5, scope.project.id=bdf9abb6d70a41acbad1d4060c21408f, user.id=dbc90e3cb55f431e91cd23da7b8958f2) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:15.899085 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a03385e-45e0-4b36-af05-a8cf5395d6c2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:15.900548 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a03385e-45e0-4b36-af05-a8cf5395d6c2 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:15.906380 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 100/196] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:19:15 2022] GET /identity/v3/role_assignments?user.id=dbc90e3cb55f431e91cd23da7b8958f2&scope.project.id=bdf9abb6d70a41acbad1d4060c21408f&role.id=4c5d52f873f94dc1b10918068376add5 => generated 608 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:15.907175 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a03385e-45e0-4b36-af05-a8cf5395d6c2 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:15.980821 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a371943d-39e6-4375-8763-e950ad38d15e None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:16.020180 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5b986c-9246-4528-a340-b9e65c45d181 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:16.021333 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5b986c-9246-4528-a340-b9e65c45d181 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:16.021664 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5b986c-9246-4528-a340-b9e65c45d181 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:16.021946 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5b986c-9246-4528-a340-b9e65c45d181 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:16.034763 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a03385e-45e0-4b36-af05-a8cf5395d6c2 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:16.041485 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:16.083466 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca1dc06-a039-403d-aae7-29bb444eba30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:16.084841 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca1dc06-a039-403d-aae7-29bb444eba30 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:16.084975 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca1dc06-a039-403d-aae7-29bb444eba30 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:16.085310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca1dc06-a039-403d-aae7-29bb444eba30 None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:16.086692 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fca1dc06-a039-403d-aae7-29bb444eba30 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:16.087254 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fca1dc06-a039-403d-aae7-29bb444eba30 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:16.089971 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fca1dc06-a039-403d-aae7-29bb444eba30 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:16.105721 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 101/197] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:19:15 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:16.107038 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca1dc06-a039-403d-aae7-29bb444eba30 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:16.115495 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5b986c-9246-4528-a340-b9e65c45d181 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:16.116987 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5b986c-9246-4528-a340-b9e65c45d181 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:16.177521 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:16.225420 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 97/198] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:19:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 12 17:19:16.226820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5b986c-9246-4528-a340-b9e65c45d181 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:16.240535 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca1dc06-a039-403d-aae7-29bb444eba30 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:16.276267 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ffd9791-4fdd-4b0c-8e01-92cbaa5018b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:16.276955 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffd9791-4fdd-4b0c-8e01-92cbaa5018b3 None admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:16.277223 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffd9791-4fdd-4b0c-8e01-92cbaa5018b3 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:16.277531 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffd9791-4fdd-4b0c-8e01-92cbaa5018b3 None admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:16.277624 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:16.278270 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ffd9791-4fdd-4b0c-8e01-92cbaa5018b3 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:16.279732 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ffd9791-4fdd-4b0c-8e01-92cbaa5018b3 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:16.335989 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5b986c-9246-4528-a340-b9e65c45d181 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:16.377155 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb2e9e4-9fec-41fe-8756-054cde0d16d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:16.377813 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb2e9e4-9fec-41fe-8756-054cde0d16d6 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:16.378105 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb2e9e4-9fec-41fe-8756-054cde0d16d6 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:16.378388 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb2e9e4-9fec-41fe-8756-054cde0d16d6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:16.406476 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 102/199] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Jul 12 17:19:16 2022] POST /identity/v3/users => generated 297 bytes in 300 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:19:16.407215 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ffd9791-4fdd-4b0c-8e01-92cbaa5018b3 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:16.478074 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb2e9e4-9fec-41fe-8756-054cde0d16d6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:16.479747 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb2e9e4-9fec-41fe-8756-054cde0d16d6 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:16.538759 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ffd9791-4fdd-4b0c-8e01-92cbaa5018b3 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:16.539202 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:16.557110 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:16.584097 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363adb56-0eaf-4dcd-bd79-0fc2e0c68126 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:16.584995 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363adb56-0eaf-4dcd-bd79-0fc2e0c68126 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:16.585279 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363adb56-0eaf-4dcd-bd79-0fc2e0c68126 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:16.585555 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363adb56-0eaf-4dcd-bd79-0fc2e0c68126 None admin] PATH_INFO: `/v3/users/84726da6e6d246e5b9500096bf044a36` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:16.586941 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363adb56-0eaf-4dcd-bd79-0fc2e0c68126 None admin] RBAC: Authorizing `identity:get_user(user_id=84726da6e6d246e5b9500096bf044a36)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:16.588377 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363adb56-0eaf-4dcd-bd79-0fc2e0c68126 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:16.590034 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 103/200] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:19:16 2022] GET /identity/v3/users/84726da6e6d246e5b9500096bf044a36 => generated 283 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:16.590851 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363adb56-0eaf-4dcd-bd79-0fc2e0c68126 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:16.618484 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 98/201] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:19:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 390 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 12 17:19:16.623944 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb2e9e4-9fec-41fe-8756-054cde0d16d6 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:16.720838 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363adb56-0eaf-4dcd-bd79-0fc2e0c68126 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:16.771807 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb2e9e4-9fec-41fe-8756-054cde0d16d6 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:16.773092 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9466024b-c674-48ca-a057-0a07349f3991 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:16.773651 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9466024b-c674-48ca-a057-0a07349f3991 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:16.773940 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9466024b-c674-48ca-a057-0a07349f3991 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:16.775012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9466024b-c674-48ca-a057-0a07349f3991 None admin] PATH_INFO: `/v3/projects/31b9ba63c2994be69d5ff224d61a9c25` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:16.775703 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9466024b-c674-48ca-a057-0a07349f3991 None admin] RBAC: Authorizing `identity:get_project(project_id=31b9ba63c2994be69d5ff224d61a9c25)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:16.777967 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9466024b-c674-48ca-a057-0a07349f3991 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:16.780447 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 104/202] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:16 2022] GET /identity/v3/projects/31b9ba63c2994be69d5ff224d61a9c25 => generated 302 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:16.785640 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9466024b-c674-48ca-a057-0a07349f3991 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:16.824191 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13fa1f81-6af8-4ddb-90b0-60b2e177c55a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:16.825314 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13fa1f81-6af8-4ddb-90b0-60b2e177c55a None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:16.825856 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13fa1f81-6af8-4ddb-90b0-60b2e177c55a None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:16.826140 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13fa1f81-6af8-4ddb-90b0-60b2e177c55a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:16.847811 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:16.858856 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:16.909147 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9466024b-c674-48ca-a057-0a07349f3991 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:16.927961 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13fa1f81-6af8-4ddb-90b0-60b2e177c55a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:16.929794 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13fa1f81-6af8-4ddb-90b0-60b2e177c55a None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:16.950126 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f4ddc7-8e33-4832-bc39-028c0a550e9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:16.951603 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f4ddc7-8e33-4832-bc39-028c0a550e9f None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:16.952093 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f4ddc7-8e33-4832-bc39-028c0a550e9f None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:16.952571 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f4ddc7-8e33-4832-bc39-028c0a550e9f None admin] PATH_INFO: `/v3/projects/4857259d2a364dff960547dc0aac75e0` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:16.954051 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f4ddc7-8e33-4832-bc39-028c0a550e9f None admin] RBAC: Authorizing `identity:get_project(project_id=4857259d2a364dff960547dc0aac75e0)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:16.956001 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f4ddc7-8e33-4832-bc39-028c0a550e9f None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:16.958585 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 105/203] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:16 2022] GET /identity/v3/projects/4857259d2a364dff960547dc0aac75e0 => generated 298 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:16.959718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f4ddc7-8e33-4832-bc39-028c0a550e9f None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:17.078247 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 99/204] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:19:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 455 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 12 17:19:17.080339 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13fa1f81-6af8-4ddb-90b0-60b2e177c55a None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:17.110473 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f4ddc7-8e33-4832-bc39-028c0a550e9f None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:17.134867 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:17.144952 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:17.151398 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3cb1c4a-6659-4e48-a3fd-99ac25e33678 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:17.152720 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3cb1c4a-6659-4e48-a3fd-99ac25e33678 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:17.153192 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3cb1c4a-6659-4e48-a3fd-99ac25e33678 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:17.153920 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3cb1c4a-6659-4e48-a3fd-99ac25e33678 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:17.155138 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3cb1c4a-6659-4e48-a3fd-99ac25e33678 None admin] RBAC: Adding query filter params (role.id=eb05bf286c894ea9bf44a6c023a4323b, scope.project.id=31b9ba63c2994be69d5ff224d61a9c25, user.id=84726da6e6d246e5b9500096bf044a36) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:17.155945 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3cb1c4a-6659-4e48-a3fd-99ac25e33678 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:17.158023 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3cb1c4a-6659-4e48-a3fd-99ac25e33678 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:17.166526 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 106/205] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:19:16 2022] GET /identity/v3/role_assignments?user.id=84726da6e6d246e5b9500096bf044a36&scope.project.id=31b9ba63c2994be69d5ff224d61a9c25&role.id=eb05bf286c894ea9bf44a6c023a4323b => generated 259 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:17.168205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3cb1c4a-6659-4e48-a3fd-99ac25e33678 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:17.210924 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13fa1f81-6af8-4ddb-90b0-60b2e177c55a None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:17.251370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b0fb55-73b9-4693-a1a6-cf4f047003dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:17.252210 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b0fb55-73b9-4693-a1a6-cf4f047003dc None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:17.252577 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b0fb55-73b9-4693-a1a6-cf4f047003dc None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:17.252933 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9b0fb55-73b9-4693-a1a6-cf4f047003dc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:17.253983 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b0fb55-73b9-4693-a1a6-cf4f047003dc None admin] RBAC: Adding query filter params (role.id=6670fba31f504efb9ae95813d9b6d660, scope.project.id=4857259d2a364dff960547dc0aac75e0, user.id=cf24bc418fc94f0f974e1870bd9bea31) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:17.254290 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b0fb55-73b9-4693-a1a6-cf4f047003dc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:17.258332 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9b0fb55-73b9-4693-a1a6-cf4f047003dc None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:17.264974 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 100/206] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:19:17 2022] GET /identity/v3/role_assignments?user.id=cf24bc418fc94f0f974e1870bd9bea31&scope.project.id=4857259d2a364dff960547dc0aac75e0&role.id=6670fba31f504efb9ae95813d9b6d660 => generated 608 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:17.311206 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3cb1c4a-6659-4e48-a3fd-99ac25e33678 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:17.354917 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6489be11-b94d-4839-931c-9d4fe9d878d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:17.356111 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6489be11-b94d-4839-931c-9d4fe9d878d2 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:17.356680 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6489be11-b94d-4839-931c-9d4fe9d878d2 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:17.357142 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6489be11-b94d-4839-931c-9d4fe9d878d2 None admin] PATH_INFO: `/v3/projects/31b9ba63c2994be69d5ff224d61a9c25` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:17.358982 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6489be11-b94d-4839-931c-9d4fe9d878d2 None admin] RBAC: Authorizing `identity:get_project(project_id=31b9ba63c2994be69d5ff224d61a9c25)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:17.360673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6489be11-b94d-4839-931c-9d4fe9d878d2 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:17.363155 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 107/207] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:17 2022] GET /identity/v3/projects/31b9ba63c2994be69d5ff224d61a9c25 => generated 302 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:17.370479 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b0fb55-73b9-4693-a1a6-cf4f047003dc None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:17.516342 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9b0fb55-73b9-4693-a1a6-cf4f047003dc None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:17.528865 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:17.556533 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-781d7539-ad3d-4ee1-8c1d-611bcbe7d1f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:17.557209 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781d7539-ad3d-4ee1-8c1d-611bcbe7d1f1 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:17.557499 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781d7539-ad3d-4ee1-8c1d-611bcbe7d1f1 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:17.557784 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781d7539-ad3d-4ee1-8c1d-611bcbe7d1f1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:17.558735 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-781d7539-ad3d-4ee1-8c1d-611bcbe7d1f1 None admin] RBAC: Adding query filter params (role.id=0c239663092d459e8c9724963e78da76, scope.project.id=31b9ba63c2994be69d5ff224d61a9c25, user.id=84726da6e6d246e5b9500096bf044a36) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:17.559058 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-781d7539-ad3d-4ee1-8c1d-611bcbe7d1f1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:17.560770 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-781d7539-ad3d-4ee1-8c1d-611bcbe7d1f1 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:17.578313 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 101/208] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:19:17 2022] GET /identity/v3/role_assignments?user.id=84726da6e6d246e5b9500096bf044a36&scope.project.id=31b9ba63c2994be69d5ff224d61a9c25&role.id=0c239663092d459e8c9724963e78da76 => generated 259 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:18.689385 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05f5c8d-fbd2-48e1-a6da-69ae3729d7d9 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:18.689886 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05f5c8d-fbd2-48e1-a6da-69ae3729d7d9 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:18.689960 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05f5c8d-fbd2-48e1-a6da-69ae3729d7d9 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:18.691350 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 108/209] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:18.697549 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919abd18-a5f8-4096-b42a-f39b950710f1 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:18.697969 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919abd18-a5f8-4096-b42a-f39b950710f1 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:18.698141 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919abd18-a5f8-4096-b42a-f39b950710f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:18.715612 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-919abd18-a5f8-4096-b42a-f39b950710f1 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:18.716974 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-919abd18-a5f8-4096-b42a-f39b950710f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:18.795016 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 102/210] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 12 17:19:18.819102 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05f5c8d-fbd2-48e1-a6da-69ae3729d7d9 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:18.874389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:18.927475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05f5c8d-fbd2-48e1-a6da-69ae3729d7d9 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:18.934451 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bb1c05-3322-4cec-9663-0e767f084bb5 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:18.935000 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bb1c05-3322-4cec-9663-0e767f084bb5 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:18.935395 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bb1c05-3322-4cec-9663-0e767f084bb5 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:18.937010 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 103/211] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 12 17:19:18.943199 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763d10f7-67f8-4e3a-9c9f-507ed6da747d None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:18.943572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763d10f7-67f8-4e3a-9c9f-507ed6da747d None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:18.944204 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763d10f7-67f8-4e3a-9c9f-507ed6da747d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:18.964901 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-763d10f7-67f8-4e3a-9c9f-507ed6da747d None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:18.966617 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-763d10f7-67f8-4e3a-9c9f-507ed6da747d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:18.969436 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27115eb2-5ff8-45cd-9f62-ff450ea9ae53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:18.970445 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27115eb2-5ff8-45cd-9f62-ff450ea9ae53 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:18.970743 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27115eb2-5ff8-45cd-9f62-ff450ea9ae53 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:18.971028 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27115eb2-5ff8-45cd-9f62-ff450ea9ae53 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:18.982805 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27115eb2-5ff8-45cd-9f62-ff450ea9ae53 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:18.990788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-27115eb2-5ff8-45cd-9f62-ff450ea9ae53 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 12 17:19:18.990788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:18.990788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:18.990788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:18.990788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:18.990788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:18.990788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:18.990788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:18.990788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:18.990788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:18.990788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:18.990788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:18.990788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:18.990788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:18.990788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:18.990788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:18.991755 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:18.991755 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:18.991755 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:18.991755 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:18.991755 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:18.991755 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:18.991755 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:18.991755 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:18.991755 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:18.991755 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:18.991755 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:18.991755 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:18.991755 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:18.991755 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:18.991755 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:18.991755 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:18.992535 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:18.992535 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:18.992535 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:18.992535 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:18.992535 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:18.992535 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:18.992535 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:18.992535 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:18.992535 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:18.992535 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:18.992535 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 12 17:19:18.992535 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:18.992535 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 109/212] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:18 2022] GET /identity/v3/roles/member => generated 84 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:19.000218 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27115eb2-5ff8-45cd-9f62-ff450ea9ae53 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:19.053685 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 104/213] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 12 17:19:19.078488 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-763d10f7-67f8-4e3a-9c9f-507ed6da747d None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:19.115392 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:19.132195 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27115eb2-5ff8-45cd-9f62-ff450ea9ae53 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:19.148054 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:19.172412 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32729a53-3852-49db-a421-4d3d335b9c5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:19.173245 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32729a53-3852-49db-a421-4d3d335b9c5a None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:19.173624 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32729a53-3852-49db-a421-4d3d335b9c5a None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:19.174026 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32729a53-3852-49db-a421-4d3d335b9c5a None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:19.174939 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32729a53-3852-49db-a421-4d3d335b9c5a None admin] RBAC: Adding query filter params (name=member) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:19.175352 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32729a53-3852-49db-a421-4d3d335b9c5a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:19.176950 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32729a53-3852-49db-a421-4d3d335b9c5a None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:19.191229 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 110/214] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Jul 12 17:19:18 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:19:19.193313 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32729a53-3852-49db-a421-4d3d335b9c5a None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:19.199041 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-763d10f7-67f8-4e3a-9c9f-507ed6da747d None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:19.245008 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece7554f-c08c-4c05-849d-1b39db765e07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:19.246577 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece7554f-c08c-4c05-849d-1b39db765e07 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:19.247112 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece7554f-c08c-4c05-849d-1b39db765e07 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:19.247531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece7554f-c08c-4c05-849d-1b39db765e07 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:19.248984 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece7554f-c08c-4c05-849d-1b39db765e07 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:19.250913 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece7554f-c08c-4c05-849d-1b39db765e07 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:19.253130 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 105/215] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:19:19 2022] GET /identity/v3/domains/default => generated 199 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:19.258810 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece7554f-c08c-4c05-849d-1b39db765e07 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:19.305519 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32729a53-3852-49db-a421-4d3d335b9c5a None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:19.343783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3fad56c-8385-4d53-91f6-cd45bc370c39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:19.344764 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fad56c-8385-4d53-91f6-cd45bc370c39 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:19.345139 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fad56c-8385-4d53-91f6-cd45bc370c39 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:19.345513 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fad56c-8385-4d53-91f6-cd45bc370c39 None admin] PATH_INFO: `/v3/users/5536e3dff3c04843adc4527073dd2e59` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:19.360877 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3fad56c-8385-4d53-91f6-cd45bc370c39 None admin] RBAC: Authorizing `identity:get_user(user_id=5536e3dff3c04843adc4527073dd2e59)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:19.362495 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3fad56c-8385-4d53-91f6-cd45bc370c39 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:19.364398 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 111/216] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:19:19 2022] GET /identity/v3/users/5536e3dff3c04843adc4527073dd2e59 => generated 297 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:19:19.366596 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece7554f-c08c-4c05-849d-1b39db765e07 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:19.368417 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:19.370621 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3fad56c-8385-4d53-91f6-cd45bc370c39 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:19.403654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:19.404713 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-655f9927-8f1e-431b-9852-cecf6669d434 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:19.405656 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655f9927-8f1e-431b-9852-cecf6669d434 None admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:19.406138 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655f9927-8f1e-431b-9852-cecf6669d434 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:19.406634 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655f9927-8f1e-431b-9852-cecf6669d434 None admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:19.408689 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-655f9927-8f1e-431b-9852-cecf6669d434 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:19.410339 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-655f9927-8f1e-431b-9852-cecf6669d434 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:19.478908 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3fad56c-8385-4d53-91f6-cd45bc370c39 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:19.506603 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 106/217] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Jul 12 17:19:19 2022] POST /identity/v3/users => generated 297 bytes in 248 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:19:19.508124 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a39a480-91d9-48bd-b847-9e3c2b6e2df8 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:19.508426 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a39a480-91d9-48bd-b847-9e3c2b6e2df8 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:19.508734 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a39a480-91d9-48bd-b847-9e3c2b6e2df8 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:19.510174 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 107/218] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:19.516349 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8391b362-63b9-4a8a-9e7e-02b052502cc2 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:19.516957 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8391b362-63b9-4a8a-9e7e-02b052502cc2 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:19.517457 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8391b362-63b9-4a8a-9e7e-02b052502cc2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:19.526470 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf99d26-8608-4b4c-8869-c5961a3d696f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:19.527507 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf99d26-8608-4b4c-8869-c5961a3d696f None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:19.528014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf99d26-8608-4b4c-8869-c5961a3d696f None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:19.528470 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf99d26-8608-4b4c-8869-c5961a3d696f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:19.536795 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-8391b362-63b9-4a8a-9e7e-02b052502cc2 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:19.538191 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-8391b362-63b9-4a8a-9e7e-02b052502cc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:19.613434 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf99d26-8608-4b4c-8869-c5961a3d696f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:19.616005 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf99d26-8608-4b4c-8869-c5961a3d696f None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:19.625599 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 108/219] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:19.648669 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8391b362-63b9-4a8a-9e7e-02b052502cc2 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:19.654102 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:19.742307 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 112/220] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:19:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 12 17:19:19.743856 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e41b607-493d-4959-9c1d-e6a3ce8e1bba None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:19.744152 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e41b607-493d-4959-9c1d-e6a3ce8e1bba None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:19.744433 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e41b607-493d-4959-9c1d-e6a3ce8e1bba None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:19.745726 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 113/221] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:19.747698 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e41b607-493d-4959-9c1d-e6a3ce8e1bba None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:19.755748 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:19.772941 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8391b362-63b9-4a8a-9e7e-02b052502cc2 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:19.813028 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caecd4b0-9f1f-4b3c-a029-d9db4ec66497 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:19.814152 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caecd4b0-9f1f-4b3c-a029-d9db4ec66497 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:19.814457 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caecd4b0-9f1f-4b3c-a029-d9db4ec66497 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:19.814813 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caecd4b0-9f1f-4b3c-a029-d9db4ec66497 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:19.831203 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caecd4b0-9f1f-4b3c-a029-d9db4ec66497 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:19.842668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-caecd4b0-9f1f-4b3c-a029-d9db4ec66497 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:19:19.842668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:19.842668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:19.842668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:19.842668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:19.842668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:19.842668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:19.842668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:19.842668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:19.842668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:19.842668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:19.842668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:19.842668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:19.842668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:19.842668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:19.842668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:19.843654 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:19.843654 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:19.843654 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:19.843654 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:19.843654 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:19.843654 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:19.843654 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:19.843654 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:19.843654 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:19.843654 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:19.843654 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:19.843654 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:19.843654 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:19.843654 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:19.843654 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:19.843654 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:19.844444 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:19.844444 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:19.844444 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:19.844444 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:19.844444 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:19.844444 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:19.844444 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:19.844444 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:19.844444 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:19.844444 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:19.844444 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:19:19.844444 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:19:19.844444 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 109/222] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Jul 12 17:19:19 2022] GET /identity/v3/roles/admin => generated 83 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:19:19.849455 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5585c3-025c-40a8-bcd2-86d575291461 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:19.849455 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5585c3-025c-40a8-bcd2-86d575291461 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:19.849455 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5585c3-025c-40a8-bcd2-86d575291461 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:19.869577 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-cf5585c3-025c-40a8-bcd2-86d575291461 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:19.870984 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-cf5585c3-025c-40a8-bcd2-86d575291461 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:19.881290 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e41b607-493d-4959-9c1d-e6a3ce8e1bba None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:19.924668 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:19.932671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981dfb9c-36e7-4762-985a-f1ddf3540337 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:19.933898 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981dfb9c-36e7-4762-985a-f1ddf3540337 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:19.934184 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981dfb9c-36e7-4762-985a-f1ddf3540337 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:19.934488 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981dfb9c-36e7-4762-985a-f1ddf3540337 None admin] PATH_INFO: `/v3/projects/31b9ba63c2994be69d5ff224d61a9c25` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:19.936262 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-981dfb9c-36e7-4762-985a-f1ddf3540337 None admin] RBAC: Authorizing `identity:get_project(project_id=31b9ba63c2994be69d5ff224d61a9c25)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:19.938066 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-981dfb9c-36e7-4762-985a-f1ddf3540337 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:19.940989 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 114/223] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:19 2022] GET /identity/v3/projects/31b9ba63c2994be69d5ff224d61a9c25 => generated 302 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:19.942499 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981dfb9c-36e7-4762-985a-f1ddf3540337 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:19.964712 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 110/224] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:19.966089 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5585c3-025c-40a8-bcd2-86d575291461 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:20.055993 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:20.074306 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981dfb9c-36e7-4762-985a-f1ddf3540337 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:20.081890 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5585c3-025c-40a8-bcd2-86d575291461 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:20.120096 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54aca763-1677-461f-adb8-ecebc0cf3179 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:20.120787 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54aca763-1677-461f-adb8-ecebc0cf3179 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:20.121071 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54aca763-1677-461f-adb8-ecebc0cf3179 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:20.121396 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54aca763-1677-461f-adb8-ecebc0cf3179 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:20.121796 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b8a8ab-630c-4f02-8094-7628362d53eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:20.122463 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54aca763-1677-461f-adb8-ecebc0cf3179 None admin] RBAC: Adding query filter params (role.id=4c5d52f873f94dc1b10918068376add5, scope.project.id=31b9ba63c2994be69d5ff224d61a9c25, user.id=5536e3dff3c04843adc4527073dd2e59) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:20.122643 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b8a8ab-630c-4f02-8094-7628362d53eb None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:20.122792 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b8a8ab-630c-4f02-8094-7628362d53eb None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:20.122936 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54aca763-1677-461f-adb8-ecebc0cf3179 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:20.123078 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b8a8ab-630c-4f02-8094-7628362d53eb None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:20.123804 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b8a8ab-630c-4f02-8094-7628362d53eb None admin] RBAC: Adding query filter params (name=admin) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:20.124121 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b8a8ab-630c-4f02-8094-7628362d53eb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:20.124282 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54aca763-1677-461f-adb8-ecebc0cf3179 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:20.125494 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b8a8ab-630c-4f02-8094-7628362d53eb None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:20.132495 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 111/225] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:19:19 2022] GET /identity/v3/role_assignments?user.id=5536e3dff3c04843adc4527073dd2e59&scope.project.id=31b9ba63c2994be69d5ff224d61a9c25&role.id=4c5d52f873f94dc1b10918068376add5 => generated 259 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:20.133173 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54aca763-1677-461f-adb8-ecebc0cf3179 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:20.136400 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 115/226] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:19 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:20.144214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b8a8ab-630c-4f02-8094-7628362d53eb None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:20.225456 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:20.279505 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b8a8ab-630c-4f02-8094-7628362d53eb None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:20.324576 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbcae1b9-8cc9-4b3c-b34b-d799bdcefe40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:20.325731 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcae1b9-8cc9-4b3c-b34b-d799bdcefe40 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:20.326836 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcae1b9-8cc9-4b3c-b34b-d799bdcefe40 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:20.327267 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcae1b9-8cc9-4b3c-b34b-d799bdcefe40 None admin] PATH_INFO: `/v3/users/84726da6e6d246e5b9500096bf044a36` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:20.328867 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbcae1b9-8cc9-4b3c-b34b-d799bdcefe40 None admin] RBAC: Authorizing `identity:get_user(user_id=84726da6e6d246e5b9500096bf044a36)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:20.330655 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbcae1b9-8cc9-4b3c-b34b-d799bdcefe40 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:20.332967 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 116/227] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:19:20 2022] GET /identity/v3/users/84726da6e6d246e5b9500096bf044a36 => generated 283 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:20.343364 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbcae1b9-8cc9-4b3c-b34b-d799bdcefe40 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:20.376743 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54aca763-1677-461f-adb8-ecebc0cf3179 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:20.415809 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5da7b6c-0faa-4540-b458-5d5f7b83ed67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:20.416835 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5da7b6c-0faa-4540-b458-5d5f7b83ed67 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:20.417129 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5da7b6c-0faa-4540-b458-5d5f7b83ed67 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:20.417413 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5da7b6c-0faa-4540-b458-5d5f7b83ed67 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:20.429968 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5da7b6c-0faa-4540-b458-5d5f7b83ed67 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:20.438955 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-f5da7b6c-0faa-4540-b458-5d5f7b83ed67 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 12 17:19:20.438955 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:20.438955 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:20.438955 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:20.438955 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:20.438955 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:20.438955 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:20.438955 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:20.438955 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:20.438955 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:20.438955 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:20.438955 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:20.438955 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:20.438955 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:20.438955 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:20.438955 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:20.439897 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:20.439897 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:20.439897 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:20.439897 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:20.439897 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:20.439897 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:20.439897 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:20.439897 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:20.439897 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:20.439897 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:20.439897 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:20.439897 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:20.439897 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:20.439897 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:20.439897 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:20.439897 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:20.440692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:20.440692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:20.440692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:20.440692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:20.440692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:20.440692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:20.440692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:20.440692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:20.440692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:20.440692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:20.440692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 12 17:19:20.440692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:19:20.440692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 112/228] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Jul 12 17:19:20 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 307 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:20.448390 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5da7b6c-0faa-4540-b458-5d5f7b83ed67 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:20.451575 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:20.561626 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5da7b6c-0faa-4540-b458-5d5f7b83ed67 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:20.618817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbcae1b9-8cc9-4b3c-b34b-d799bdcefe40 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:20.650047 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:20.658256 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb85658-c698-4287-848e-8479bafce5cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:20.658975 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb85658-c698-4287-848e-8479bafce5cb None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:20.665424 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb85658-c698-4287-848e-8479bafce5cb None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:20.665424 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb85658-c698-4287-848e-8479bafce5cb None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:20.665424 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb85658-c698-4287-848e-8479bafce5cb None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:20.665424 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb85658-c698-4287-848e-8479bafce5cb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:20.665424 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb85658-c698-4287-848e-8479bafce5cb None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:20.665839 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d587d9d-87d0-486f-88f7-1f574c65b856 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:20.665839 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d587d9d-87d0-486f-88f7-1f574c65b856 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:20.665981 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d587d9d-87d0-486f-88f7-1f574c65b856 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:20.666194 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d587d9d-87d0-486f-88f7-1f574c65b856 None admin] PATH_INFO: `/v3/projects/31b9ba63c2994be69d5ff224d61a9c25` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:20.667400 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d587d9d-87d0-486f-88f7-1f574c65b856 None admin] RBAC: Authorizing `identity:get_project(project_id=31b9ba63c2994be69d5ff224d61a9c25)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:20.668904 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d587d9d-87d0-486f-88f7-1f574c65b856 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:20.670978 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 117/229] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:20 2022] GET /identity/v3/projects/31b9ba63c2994be69d5ff224d61a9c25 => generated 302 bytes in 329 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:19:20.676785 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d587d9d-87d0-486f-88f7-1f574c65b856 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:20.679323 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 113/230] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:19:20 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:20.694606 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb85658-c698-4287-848e-8479bafce5cb None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:20.795041 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d587d9d-87d0-486f-88f7-1f574c65b856 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:20.826689 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:20.858394 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2447ff-bd29-4c77-985f-befc21d82bc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:20.860260 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2447ff-bd29-4c77-985f-befc21d82bc2 None admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:20.860701 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2447ff-bd29-4c77-985f-befc21d82bc2 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:20.861155 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2447ff-bd29-4c77-985f-befc21d82bc2 None admin] PATH_INFO: `/v3/projects/31b9ba63c2994be69d5ff224d61a9c25/users/84726da6e6d246e5b9500096bf044a36/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:20.864322 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2447ff-bd29-4c77-985f-befc21d82bc2 None admin] RBAC: Authorizing `identity:create_grant(project_id=31b9ba63c2994be69d5ff224d61a9c25, user_id=84726da6e6d246e5b9500096bf044a36, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:20.866238 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2447ff-bd29-4c77-985f-befc21d82bc2 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:20.879696 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb85658-c698-4287-848e-8479bafce5cb None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:20.918142 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08945be4-b5ce-4669-bba3-ecad2ad66cb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:20.919588 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08945be4-b5ce-4669-bba3-ecad2ad66cb4 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:20.920016 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08945be4-b5ce-4669-bba3-ecad2ad66cb4 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:20.920493 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08945be4-b5ce-4669-bba3-ecad2ad66cb4 None admin] PATH_INFO: `/v3/users/84726da6e6d246e5b9500096bf044a36` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:20.921930 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08945be4-b5ce-4669-bba3-ecad2ad66cb4 None admin] RBAC: Authorizing `identity:get_user(user_id=84726da6e6d246e5b9500096bf044a36)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:20.923573 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08945be4-b5ce-4669-bba3-ecad2ad66cb4 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:20.925755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 114/231] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:19:20 2022] GET /identity/v3/users/84726da6e6d246e5b9500096bf044a36 => generated 283 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:20.935518 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08945be4-b5ce-4669-bba3-ecad2ad66cb4 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:20.953152 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 118/232] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Jul 12 17:19:20 2022] PUT /identity/v3/projects/31b9ba63c2994be69d5ff224d61a9c25/users/84726da6e6d246e5b9500096bf044a36/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:19:21.076160 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08945be4-b5ce-4669-bba3-ecad2ad66cb4 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:21.111137 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:21.121439 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae70e333-0070-43b4-97b3-b935e5a3969d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:21.122350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae70e333-0070-43b4-97b3-b935e5a3969d None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:21.122668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae70e333-0070-43b4-97b3-b935e5a3969d None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:21.122950 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae70e333-0070-43b4-97b3-b935e5a3969d None admin] PATH_INFO: `/v3/projects/31b9ba63c2994be69d5ff224d61a9c25` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:21.125362 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae70e333-0070-43b4-97b3-b935e5a3969d None admin] RBAC: Authorizing `identity:get_project(project_id=31b9ba63c2994be69d5ff224d61a9c25)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:21.126976 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae70e333-0070-43b4-97b3-b935e5a3969d None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:21.129320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 115/233] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:20 2022] GET /identity/v3/projects/31b9ba63c2994be69d5ff224d61a9c25 => generated 302 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:21.135649 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2447ff-bd29-4c77-985f-befc21d82bc2 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:21.203737 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:21.253897 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2447ff-bd29-4c77-985f-befc21d82bc2 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:21.293397 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba74b08-a627-4234-9f6b-46b209163259 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:21.294528 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba74b08-a627-4234-9f6b-46b209163259 None admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:21.294824 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba74b08-a627-4234-9f6b-46b209163259 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:21.295108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba74b08-a627-4234-9f6b-46b209163259 None admin] PATH_INFO: `/v3/projects/31b9ba63c2994be69d5ff224d61a9c25/users/84726da6e6d246e5b9500096bf044a36/roles/0c239663092d459e8c9724963e78da76` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:21.297354 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bba74b08-a627-4234-9f6b-46b209163259 None admin] RBAC: Authorizing `identity:create_grant(project_id=31b9ba63c2994be69d5ff224d61a9c25, user_id=84726da6e6d246e5b9500096bf044a36, role_id=0c239663092d459e8c9724963e78da76)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:21.298945 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bba74b08-a627-4234-9f6b-46b209163259 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:21.430002 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 119/234] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Jul 12 17:19:21 2022] PUT /identity/v3/projects/31b9ba63c2994be69d5ff224d61a9c25/users/84726da6e6d246e5b9500096bf044a36/roles/0c239663092d459e8c9724963e78da76 => generated 0 bytes in 295 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:19:22.085277 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a93e5e-cc3d-4f6b-bee5-4437b49a5a85 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:22.085798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a93e5e-cc3d-4f6b-bee5-4437b49a5a85 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:22.086093 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27a93e5e-cc3d-4f6b-bee5-4437b49a5a85 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:22.087809 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 116/235] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:22.100568 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032b2911-3712-478d-8b4b-623d1a66a7bb None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:22.101060 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032b2911-3712-478d-8b4b-623d1a66a7bb None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:22.101135 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032b2911-3712-478d-8b4b-623d1a66a7bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:22.123164 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-032b2911-3712-478d-8b4b-623d1a66a7bb None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:22.124399 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-032b2911-3712-478d-8b4b-623d1a66a7bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:22.213098 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 120/236] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:22.248530 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27a93e5e-cc3d-4f6b-bee5-4437b49a5a85 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:22.250927 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9088d2-c50f-4692-8f24-acabf5c59f34 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:22.251228 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9088d2-c50f-4692-8f24-acabf5c59f34 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:22.251514 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9088d2-c50f-4692-8f24-acabf5c59f34 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:22.252959 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 121/237] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:22.259995 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d89fac-14fd-4091-9d87-11c3cb4f3446 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:22.260591 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d89fac-14fd-4091-9d87-11c3cb4f3446 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:22.261078 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d89fac-14fd-4091-9d87-11c3cb4f3446 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:22.291322 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-b8d89fac-14fd-4091-9d87-11c3cb4f3446 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:22.294485 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-b8d89fac-14fd-4091-9d87-11c3cb4f3446 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:22.365467 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:22.394098 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 122/238] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 12 17:19:22.430721 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27a93e5e-cc3d-4f6b-bee5-4437b49a5a85 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:22.447516 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d89fac-14fd-4091-9d87-11c3cb4f3446 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:22.500146 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43f4395-3bd8-4efb-81c0-79ca64c11a8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:22.501243 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43f4395-3bd8-4efb-81c0-79ca64c11a8a None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:22.501525 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43f4395-3bd8-4efb-81c0-79ca64c11a8a None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:22.501799 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43f4395-3bd8-4efb-81c0-79ca64c11a8a None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:22.516647 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f43f4395-3bd8-4efb-81c0-79ca64c11a8a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:22.532743 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-f43f4395-3bd8-4efb-81c0-79ca64c11a8a None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 12 17:19:22.532743 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:22.532743 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:22.532743 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:22.532743 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:22.532743 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:22.532743 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:22.532743 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:22.532743 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:22.532743 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:22.532743 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:22.532743 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:22.532743 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:22.532743 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:22.532743 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:22.532743 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:22.533775 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:22.533775 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:22.533775 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:22.533775 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:22.533775 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:22.533775 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:22.533775 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:22.533775 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:22.533775 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:22.533775 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:22.533775 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:22.533775 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:22.533775 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:22.533775 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:22.533775 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:22.533775 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:22.534611 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:22.534611 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:22.534611 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:22.534611 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:22.534611 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:22.534611 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:22.534611 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:22.534611 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:22.534611 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:22.534611 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:22.534611 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 12 17:19:22.534611 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:19:22.535277 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 117/239] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:22 2022] GET /identity/v3/roles/reader => generated 84 bytes in 286 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:22.539677 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43f4395-3bd8-4efb-81c0-79ca64c11a8a None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:22.576809 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d89fac-14fd-4091-9d87-11c3cb4f3446 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:22.624882 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd5938b-779d-4f43-8c6b-659cca273cb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:22.626017 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd5938b-779d-4f43-8c6b-659cca273cb8 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:22.626830 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd5938b-779d-4f43-8c6b-659cca273cb8 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:22.628131 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:22.628735 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd5938b-779d-4f43-8c6b-659cca273cb8 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:22.632709 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfd5938b-779d-4f43-8c6b-659cca273cb8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:22.635297 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfd5938b-779d-4f43-8c6b-659cca273cb8 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:22.638184 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 123/240] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:19:22 2022] GET /identity/v3/domains/default => generated 199 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:19:22.640068 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c01e6c8-7764-491c-8c32-244f02280dc0 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:22.646880 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c01e6c8-7764-491c-8c32-244f02280dc0 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:22.646880 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c01e6c8-7764-491c-8c32-244f02280dc0 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:22.647190 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 124/241] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:22 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:22.648793 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c01e6c8-7764-491c-8c32-244f02280dc0 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:22.687495 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43f4395-3bd8-4efb-81c0-79ca64c11a8a None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:22.703848 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:22.739579 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d0e4042-2ddb-4505-9f4e-6488e41a01e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:22.740728 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0e4042-2ddb-4505-9f4e-6488e41a01e9 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:22.741198 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0e4042-2ddb-4505-9f4e-6488e41a01e9 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:22.741644 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0e4042-2ddb-4505-9f4e-6488e41a01e9 None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:22.743201 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d0e4042-2ddb-4505-9f4e-6488e41a01e9 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:22.743703 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d0e4042-2ddb-4505-9f4e-6488e41a01e9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:22.745733 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d0e4042-2ddb-4505-9f4e-6488e41a01e9 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:22.763908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 118/242] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Jul 12 17:19:22 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:19:22.764615 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6040039-aeaa-414e-a276-7738056248e4 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:22.764720 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6040039-aeaa-414e-a276-7738056248e4 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:22.765116 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6040039-aeaa-414e-a276-7738056248e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:22.781969 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c01e6c8-7764-491c-8c32-244f02280dc0 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:22.783769 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-c6040039-aeaa-414e-a276-7738056248e4 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:22.786336 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-c6040039-aeaa-414e-a276-7738056248e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:22.836834 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36bde790-b25e-4656-a67a-a7421cceda89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:22.837936 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bde790-b25e-4656-a67a-a7421cceda89 None admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:22.838460 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bde790-b25e-4656-a67a-a7421cceda89 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:22.838939 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bde790-b25e-4656-a67a-a7421cceda89 None admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:22.839859 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36bde790-b25e-4656-a67a-a7421cceda89 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:22.842189 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36bde790-b25e-4656-a67a-a7421cceda89 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:22.878578 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 119/243] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:22.878578 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6040039-aeaa-414e-a276-7738056248e4 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:22.932681 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 125/244] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Jul 12 17:19:22 2022] POST /identity/v3/users => generated 293 bytes in 285 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:19:22.934178 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36bde790-b25e-4656-a67a-a7421cceda89 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:22.937224 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:22.994197 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6040039-aeaa-414e-a276-7738056248e4 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:23.038997 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee48b8f8-cdc5-4bf0-acc8-8d3960163192 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:23.039706 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee48b8f8-cdc5-4bf0-acc8-8d3960163192 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:23.039995 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee48b8f8-cdc5-4bf0-acc8-8d3960163192 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:23.040330 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee48b8f8-cdc5-4bf0-acc8-8d3960163192 None admin] PATH_INFO: `/v3/users/337d5991159749c8bab7a94ad521c19e` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:23.059340 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee48b8f8-cdc5-4bf0-acc8-8d3960163192 None admin] RBAC: Authorizing `identity:get_user(user_id=337d5991159749c8bab7a94ad521c19e)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:23.061012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee48b8f8-cdc5-4bf0-acc8-8d3960163192 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:23.063052 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 120/245] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:19:22 2022] GET /identity/v3/users/337d5991159749c8bab7a94ad521c19e => generated 297 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:23.069613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee48b8f8-cdc5-4bf0-acc8-8d3960163192 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:23.085283 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:23.088939 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36bde790-b25e-4656-a67a-a7421cceda89 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:23.135900 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebade831-c5a8-4b27-b9c3-41e86258408d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:23.137491 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebade831-c5a8-4b27-b9c3-41e86258408d None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:23.137977 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebade831-c5a8-4b27-b9c3-41e86258408d None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:23.138471 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebade831-c5a8-4b27-b9c3-41e86258408d None admin] PATH_INFO: `/v3/roles/member` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:23.156883 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebade831-c5a8-4b27-b9c3-41e86258408d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:23.165610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-ebade831-c5a8-4b27-b9c3-41e86258408d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 12 17:19:23.165610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:23.165610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:23.165610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:23.165610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:23.165610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:23.165610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:23.165610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:23.165610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:23.165610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:23.165610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:23.165610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:23.165610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:23.165610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:23.165610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:23.165610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:23.166689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:23.166689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:23.166689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:23.166689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:23.166689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:23.166689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:23.166689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:23.166689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:23.166689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:23.166689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:23.166689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:23.166689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:23.166689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:23.166689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:23.166689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:23.166689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:23.167500 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:23.167500 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:23.167500 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:23.167500 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:23.167500 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:23.167500 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:23.167500 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:23.167500 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:23.167500 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:23.167500 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:23.167500 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 12 17:19:23.167500 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:23.174576 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 126/246] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:22 2022] GET /identity/v3/roles/member => generated 84 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:23.177508 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebade831-c5a8-4b27-b9c3-41e86258408d None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:23.189498 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee48b8f8-cdc5-4bf0-acc8-8d3960163192 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:23.203673 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:23.231429 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c338a0a4-b1a7-4d97-a4cf-c39a35ec4198 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:23.232345 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c338a0a4-b1a7-4d97-a4cf-c39a35ec4198 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:23.233074 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c338a0a4-b1a7-4d97-a4cf-c39a35ec4198 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:23.233911 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c338a0a4-b1a7-4d97-a4cf-c39a35ec4198 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:23.289831 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebade831-c5a8-4b27-b9c3-41e86258408d None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:23.313432 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c338a0a4-b1a7-4d97-a4cf-c39a35ec4198 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:23.315167 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c338a0a4-b1a7-4d97-a4cf-c39a35ec4198 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:23.325706 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab674829-b255-48c3-9b34-e1a140b2bb56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:23.326721 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab674829-b255-48c3-9b34-e1a140b2bb56 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:23.327296 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab674829-b255-48c3-9b34-e1a140b2bb56 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:23.327766 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab674829-b255-48c3-9b34-e1a140b2bb56 None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:23.328707 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab674829-b255-48c3-9b34-e1a140b2bb56 None admin] RBAC: Adding query filter params (name=member) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:23.329214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab674829-b255-48c3-9b34-e1a140b2bb56 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:23.330816 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab674829-b255-48c3-9b34-e1a140b2bb56 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:23.341113 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 127/247] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Jul 12 17:19:23 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:23.347044 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab674829-b255-48c3-9b34-e1a140b2bb56 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:23.375473 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:23.436411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 121/248] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:19:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 12 17:19:23.437663 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3da20bf-1d5f-4ac7-9f4f-7a24732ec447 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:23.437954 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3da20bf-1d5f-4ac7-9f4f-7a24732ec447 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:23.438235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3da20bf-1d5f-4ac7-9f4f-7a24732ec447 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:23.445247 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 122/249] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:23 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:23.445860 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3da20bf-1d5f-4ac7-9f4f-7a24732ec447 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:23.460649 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:23.477700 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab674829-b255-48c3-9b34-e1a140b2bb56 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:23.524191 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97ef507-2ded-4ba6-bb54-86efcea59d55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:23.525185 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97ef507-2ded-4ba6-bb54-86efcea59d55 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:23.525417 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97ef507-2ded-4ba6-bb54-86efcea59d55 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:23.525703 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97ef507-2ded-4ba6-bb54-86efcea59d55 None admin] PATH_INFO: `/v3/users/5536e3dff3c04843adc4527073dd2e59` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:23.527086 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97ef507-2ded-4ba6-bb54-86efcea59d55 None admin] RBAC: Authorizing `identity:get_user(user_id=5536e3dff3c04843adc4527073dd2e59)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:23.528557 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97ef507-2ded-4ba6-bb54-86efcea59d55 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:23.530249 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 128/250] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:19:23 2022] GET /identity/v3/users/5536e3dff3c04843adc4527073dd2e59 => generated 297 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:23.531499 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14784e62-86d9-422e-be4e-09155b390141 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:23.531792 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14784e62-86d9-422e-be4e-09155b390141 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:23.532076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14784e62-86d9-422e-be4e-09155b390141 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:23.551697 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-14784e62-86d9-422e-be4e-09155b390141 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:23.552911 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-14784e62-86d9-422e-be4e-09155b390141 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:23.593205 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3da20bf-1d5f-4ac7-9f4f-7a24732ec447 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:23.635498 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b303c8f-5782-4f3e-a64d-6a1ad3dd897c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:23.636842 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b303c8f-5782-4f3e-a64d-6a1ad3dd897c None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:23.637327 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b303c8f-5782-4f3e-a64d-6a1ad3dd897c None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:23.637818 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b303c8f-5782-4f3e-a64d-6a1ad3dd897c None admin] PATH_INFO: `/v3/projects/31b9ba63c2994be69d5ff224d61a9c25` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:23.639388 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b303c8f-5782-4f3e-a64d-6a1ad3dd897c None admin] RBAC: Authorizing `identity:get_project(project_id=31b9ba63c2994be69d5ff224d61a9c25)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:23.641100 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b303c8f-5782-4f3e-a64d-6a1ad3dd897c None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:23.643139 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 123/251] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:23 2022] GET /identity/v3/projects/31b9ba63c2994be69d5ff224d61a9c25 => generated 302 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:23.644139 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b303c8f-5782-4f3e-a64d-6a1ad3dd897c None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:23.650338 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 129/252] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:23.651323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14784e62-86d9-422e-be4e-09155b390141 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:23.690915 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:23.749108 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:23.779297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b303c8f-5782-4f3e-a64d-6a1ad3dd897c None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:23.791956 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14784e62-86d9-422e-be4e-09155b390141 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:23.823607 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-781f068a-3fcc-4379-8ffc-79c401a73927 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:23.824611 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781f068a-3fcc-4379-8ffc-79c401a73927 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:23.824896 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781f068a-3fcc-4379-8ffc-79c401a73927 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:23.825175 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-781f068a-3fcc-4379-8ffc-79c401a73927 None admin] PATH_INFO: `/v3/projects/31b9ba63c2994be69d5ff224d61a9c25` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:23.826371 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-781f068a-3fcc-4379-8ffc-79c401a73927 None admin] RBAC: Authorizing `identity:get_project(project_id=31b9ba63c2994be69d5ff224d61a9c25)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:23.828138 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-781f068a-3fcc-4379-8ffc-79c401a73927 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:23.830537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 124/253] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:23 2022] GET /identity/v3/projects/31b9ba63c2994be69d5ff224d61a9c25 => generated 302 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:23.831328 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-781f068a-3fcc-4379-8ffc-79c401a73927 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:23.882532 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3834b0-6144-406c-9519-ff2b79e92f19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:23.883557 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3834b0-6144-406c-9519-ff2b79e92f19 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:23.884043 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3834b0-6144-406c-9519-ff2b79e92f19 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:23.884518 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3834b0-6144-406c-9519-ff2b79e92f19 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:23.885555 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f3834b0-6144-406c-9519-ff2b79e92f19 None admin] RBAC: Adding query filter params (role.id=6670fba31f504efb9ae95813d9b6d660, scope.project.id=31b9ba63c2994be69d5ff224d61a9c25, user.id=337d5991159749c8bab7a94ad521c19e) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:23.886093 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f3834b0-6144-406c-9519-ff2b79e92f19 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:23.887763 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f3834b0-6144-406c-9519-ff2b79e92f19 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:23.904416 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 130/254] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:19:23 2022] GET /identity/v3/role_assignments?user.id=337d5991159749c8bab7a94ad521c19e&scope.project.id=31b9ba63c2994be69d5ff224d61a9c25&role.id=6670fba31f504efb9ae95813d9b6d660 => generated 259 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:23.905573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3834b0-6144-406c-9519-ff2b79e92f19 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:23.953425 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-781f068a-3fcc-4379-8ffc-79c401a73927 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:23.998044 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f827ae3c-f2f6-4a2e-b8d1-2c166aa53b31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:23.998891 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f827ae3c-f2f6-4a2e-b8d1-2c166aa53b31 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:23.999178 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f827ae3c-f2f6-4a2e-b8d1-2c166aa53b31 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:23.999457 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f827ae3c-f2f6-4a2e-b8d1-2c166aa53b31 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:24.014810 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f827ae3c-f2f6-4a2e-b8d1-2c166aa53b31 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:24.024103 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-f827ae3c-f2f6-4a2e-b8d1-2c166aa53b31 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:19:24.024103 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:24.024103 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:24.024103 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:24.024103 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:24.024103 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:24.024103 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:24.024103 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:24.024103 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:24.024103 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:24.024103 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:24.024103 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:24.024103 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:24.024103 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:24.024103 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:24.024103 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:24.025040 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:24.025040 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:24.025040 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:24.025040 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:24.025040 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:24.025040 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:24.025040 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:24.025040 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:24.025040 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:24.025040 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:24.025040 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:24.025040 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:24.025040 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:24.025040 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:24.025040 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:24.025040 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:24.025850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:24.025850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:24.025850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:24.025850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:24.025850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:24.025850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:24.025850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:24.025850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:24.025850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:24.025850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:24.025850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:19:24.025850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:19:24.025850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 125/255] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Jul 12 17:19:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:24.031514 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f827ae3c-f2f6-4a2e-b8d1-2c166aa53b31 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:24.043737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:24.107185 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:24.148355 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3834b0-6144-406c-9519-ff2b79e92f19 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:24.166908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f827ae3c-f2f6-4a2e-b8d1-2c166aa53b31 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:24.209400 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18db3dee-b353-4dad-8d35-9915ec091fd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:24.210896 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18db3dee-b353-4dad-8d35-9915ec091fd7 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:24.211514 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18db3dee-b353-4dad-8d35-9915ec091fd7 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:24.211987 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18db3dee-b353-4dad-8d35-9915ec091fd7 None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:24.213116 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18db3dee-b353-4dad-8d35-9915ec091fd7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:24.213624 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18db3dee-b353-4dad-8d35-9915ec091fd7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:24.215795 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18db3dee-b353-4dad-8d35-9915ec091fd7 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:24.224522 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eae2abf-4604-4968-bd18-723301db647a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:24.225347 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eae2abf-4604-4968-bd18-723301db647a None admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:24.225633 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eae2abf-4604-4968-bd18-723301db647a None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:24.225913 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eae2abf-4604-4968-bd18-723301db647a None admin] PATH_INFO: `/v3/projects/31b9ba63c2994be69d5ff224d61a9c25/users/5536e3dff3c04843adc4527073dd2e59/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:24.229200 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 126/256] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:24 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:24.233190 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eae2abf-4604-4968-bd18-723301db647a None admin] RBAC: Authorizing `identity:create_grant(project_id=31b9ba63c2994be69d5ff224d61a9c25, user_id=5536e3dff3c04843adc4527073dd2e59, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:24.233190 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eae2abf-4604-4968-bd18-723301db647a None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:24.234613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18db3dee-b353-4dad-8d35-9915ec091fd7 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:24.308205 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:24.343206 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 131/257] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Jul 12 17:19:23 2022] PUT /identity/v3/projects/31b9ba63c2994be69d5ff224d61a9c25/users/5536e3dff3c04843adc4527073dd2e59/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 438 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:19:24.344616 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ddce7de-4136-47a3-86d4-93c8cd205fcd None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:24.344917 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ddce7de-4136-47a3-86d4-93c8cd205fcd None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:24.345200 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ddce7de-4136-47a3-86d4-93c8cd205fcd None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:24.355264 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 132/258] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:24 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:24.364567 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18db3dee-b353-4dad-8d35-9915ec091fd7 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:24.371601 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662efce9-448c-4cab-a1df-fdcc2ac11683 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:24.371894 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662efce9-448c-4cab-a1df-fdcc2ac11683 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:24.372180 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-662efce9-448c-4cab-a1df-fdcc2ac11683 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:24.401946 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-662efce9-448c-4cab-a1df-fdcc2ac11683 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:24.403205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-662efce9-448c-4cab-a1df-fdcc2ac11683 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:24.484236 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96951694-0711-4211-adeb-431ac7c2305d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:24.485142 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96951694-0711-4211-adeb-431ac7c2305d None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:24.485458 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96951694-0711-4211-adeb-431ac7c2305d None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:24.485746 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96951694-0711-4211-adeb-431ac7c2305d None admin] PATH_INFO: `/v3/users/84726da6e6d246e5b9500096bf044a36` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:24.489382 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96951694-0711-4211-adeb-431ac7c2305d None admin] RBAC: Authorizing `identity:get_user(user_id=84726da6e6d246e5b9500096bf044a36)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:24.490980 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96951694-0711-4211-adeb-431ac7c2305d None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:24.492715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 127/259] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:19:24 2022] GET /identity/v3/users/84726da6e6d246e5b9500096bf044a36 => generated 283 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:24.505990 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96951694-0711-4211-adeb-431ac7c2305d None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:24.579878 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 133/260] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 12 17:19:24.651408 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662efce9-448c-4cab-a1df-fdcc2ac11683 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:24.740626 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96951694-0711-4211-adeb-431ac7c2305d None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:24.780347 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:24.810529 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-662efce9-448c-4cab-a1df-fdcc2ac11683 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:24.831773 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a0673a-e92f-4276-a25c-633bdfaee423 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:24.833067 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a0673a-e92f-4276-a25c-633bdfaee423 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:24.833461 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a0673a-e92f-4276-a25c-633bdfaee423 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:24.833749 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a0673a-e92f-4276-a25c-633bdfaee423 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:24.846855 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:24.890170 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3576bd3-9d15-4a55-9b8f-90ada2cbf49d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:24.891047 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3576bd3-9d15-4a55-9b8f-90ada2cbf49d None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:24.891335 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3576bd3-9d15-4a55-9b8f-90ada2cbf49d None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:24.891618 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3576bd3-9d15-4a55-9b8f-90ada2cbf49d None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:24.924150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3576bd3-9d15-4a55-9b8f-90ada2cbf49d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:24.941150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-d3576bd3-9d15-4a55-9b8f-90ada2cbf49d None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 12 17:19:24.941150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:24.941150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:24.941150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:24.941150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:24.941150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:24.941150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:24.941150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:24.941150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:24.941150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:24.941150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:24.941150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:24.941150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:24.941150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:24.941150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:24.941150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:24.942945 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:24.942945 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:24.942945 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:24.942945 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:24.942945 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:24.942945 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:24.942945 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:24.942945 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:24.942945 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:24.942945 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:24.942945 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:24.942945 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:24.942945 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:24.942945 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:24.942945 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:24.942945 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:24.943725 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:24.943725 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:24.943725 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:24.943725 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:24.943725 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:24.943725 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:24.943725 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:24.943725 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:24.943725 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:24.943725 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:24.943725 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 12 17:19:24.943725 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:24.951546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 134/261] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Jul 12 17:19:24 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 301 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:19:24.958905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3576bd3-9d15-4a55-9b8f-90ada2cbf49d None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:24.998056 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a0673a-e92f-4276-a25c-633bdfaee423 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:24.999861 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a0673a-e92f-4276-a25c-633bdfaee423 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:25.196685 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3576bd3-9d15-4a55-9b8f-90ada2cbf49d None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:25.202212 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 128/262] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:19:24 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 697 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 12 17:19:25.214586 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a0673a-e92f-4276-a25c-633bdfaee423 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:25.255223 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e1a1075-5250-4873-b589-39fe020bc593 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:25.255841 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1a1075-5250-4873-b589-39fe020bc593 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:25.256137 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1a1075-5250-4873-b589-39fe020bc593 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:25.256427 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1a1075-5250-4873-b589-39fe020bc593 None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:25.257208 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e1a1075-5250-4873-b589-39fe020bc593 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:25.257528 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e1a1075-5250-4873-b589-39fe020bc593 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:25.259949 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e1a1075-5250-4873-b589-39fe020bc593 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:25.260566 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:25.267906 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 135/263] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:19:24 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 309 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:19:25.275010 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e1a1075-5250-4873-b589-39fe020bc593 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:25.302810 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:25.477919 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a0673a-e92f-4276-a25c-633bdfaee423 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:25.478330 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e1a1075-5250-4873-b589-39fe020bc593 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:25.554286 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36062287-279c-4e84-9056-724e737a57f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:25.555108 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36062287-279c-4e84-9056-724e737a57f9 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:25.555397 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36062287-279c-4e84-9056-724e737a57f9 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:25.555679 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36062287-279c-4e84-9056-724e737a57f9 None admin] PATH_INFO: `/v3/projects/31b9ba63c2994be69d5ff224d61a9c25` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:25.560415 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36062287-279c-4e84-9056-724e737a57f9 None admin] RBAC: Authorizing `identity:get_project(project_id=31b9ba63c2994be69d5ff224d61a9c25)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:25.561828 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36062287-279c-4e84-9056-724e737a57f9 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:25.567066 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 129/264] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:25 2022] GET /identity/v3/projects/31b9ba63c2994be69d5ff224d61a9c25 => generated 302 bytes in 350 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:25.572349 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36062287-279c-4e84-9056-724e737a57f9 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:25.572471 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46baed3c-811a-484e-b717-1cc3ceff6c87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:25.572471 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46baed3c-811a-484e-b717-1cc3ceff6c87 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:25.572471 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46baed3c-811a-484e-b717-1cc3ceff6c87 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:25.572471 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46baed3c-811a-484e-b717-1cc3ceff6c87 None admin] PATH_INFO: `/v3/users/84726da6e6d246e5b9500096bf044a36` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:25.573116 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46baed3c-811a-484e-b717-1cc3ceff6c87 None admin] RBAC: Authorizing `identity:get_user(user_id=84726da6e6d246e5b9500096bf044a36)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:25.576098 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46baed3c-811a-484e-b717-1cc3ceff6c87 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:25.576307 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 136/265] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:19:25 2022] GET /identity/v3/users/84726da6e6d246e5b9500096bf044a36 => generated 283 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:19:25.587478 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46baed3c-811a-484e-b717-1cc3ceff6c87 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:25.725157 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:25.732382 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36062287-279c-4e84-9056-724e737a57f9 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:25.734959 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:25.777567 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b82768f-208f-4f14-933a-9fbb812b7ce0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:25.778612 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b82768f-208f-4f14-933a-9fbb812b7ce0 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:25.779820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b82768f-208f-4f14-933a-9fbb812b7ce0 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:25.780329 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b82768f-208f-4f14-933a-9fbb812b7ce0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:25.781923 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b82768f-208f-4f14-933a-9fbb812b7ce0 None admin] RBAC: Adding query filter params (role.id=eb05bf286c894ea9bf44a6c023a4323b, scope.project.id=31b9ba63c2994be69d5ff224d61a9c25, user.id=84726da6e6d246e5b9500096bf044a36) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:25.782630 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b82768f-208f-4f14-933a-9fbb812b7ce0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:25.784630 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46baed3c-811a-484e-b717-1cc3ceff6c87 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:25.791274 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b82768f-208f-4f14-933a-9fbb812b7ce0 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:25.805781 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 130/266] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:19:25 2022] GET /identity/v3/role_assignments?user.id=84726da6e6d246e5b9500096bf044a36&scope.project.id=31b9ba63c2994be69d5ff224d61a9c25&role.id=eb05bf286c894ea9bf44a6c023a4323b => generated 608 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:19:25.857200 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49fbdb3-1377-4eb0-967a-0c45960ce26a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:25.858256 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49fbdb3-1377-4eb0-967a-0c45960ce26a None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:25.862876 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49fbdb3-1377-4eb0-967a-0c45960ce26a None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:25.864044 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49fbdb3-1377-4eb0-967a-0c45960ce26a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:25.972704 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b49fbdb3-1377-4eb0-967a-0c45960ce26a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:25.974256 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b49fbdb3-1377-4eb0-967a-0c45960ce26a None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:26.062931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:26.077014 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196766fc-6552-4bfb-815e-16fb3ef72b1a None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:26.077652 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196766fc-6552-4bfb-815e-16fb3ef72b1a None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:26.078132 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196766fc-6552-4bfb-815e-16fb3ef72b1a None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:26.079771 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 131/267] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 12 17:19:26.086187 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e830416-6d9d-46c3-bc0e-83836869a458 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:26.086773 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e830416-6d9d-46c3-bc0e-83836869a458 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:26.087293 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e830416-6d9d-46c3-bc0e-83836869a458 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:26.124761 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 137/268] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:19:25 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 538 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 12 17:19:26.125784 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-2e830416-6d9d-46c3-bc0e-83836869a458 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:26.127002 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-2e830416-6d9d-46c3-bc0e-83836869a458 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:26.131016 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49fbdb3-1377-4eb0-967a-0c45960ce26a None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:26.266371 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 132/269] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 12 17:19:26.268500 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f5cc47-7d2f-4987-801f-7a6849fd3a65 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:26.269009 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f5cc47-7d2f-4987-801f-7a6849fd3a65 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:26.270012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f5cc47-7d2f-4987-801f-7a6849fd3a65 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:26.272906 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 133/270] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:26.276947 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b65460-4eb0-43a2-8e9b-1308358a3c59 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:26.277334 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b65460-4eb0-43a2-8e9b-1308358a3c59 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:26.277697 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b65460-4eb0-43a2-8e9b-1308358a3c59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:26.297763 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-31b65460-4eb0-43a2-8e9b-1308358a3c59 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:26.300097 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-31b65460-4eb0-43a2-8e9b-1308358a3c59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:26.314816 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49fbdb3-1377-4eb0-967a-0c45960ce26a None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:26.368483 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f200ba-70d7-46c9-9562-591cd6451f70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:26.370316 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f200ba-70d7-46c9-9562-591cd6451f70 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:26.370803 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f200ba-70d7-46c9-9562-591cd6451f70 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:26.371497 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f200ba-70d7-46c9-9562-591cd6451f70 None admin] PATH_INFO: `/v3/projects/31b9ba63c2994be69d5ff224d61a9c25` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:26.373138 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f200ba-70d7-46c9-9562-591cd6451f70 None admin] RBAC: Authorizing `identity:get_project(project_id=31b9ba63c2994be69d5ff224d61a9c25)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:26.375277 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f200ba-70d7-46c9-9562-591cd6451f70 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:26.377769 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 138/271] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:26 2022] GET /identity/v3/projects/31b9ba63c2994be69d5ff224d61a9c25 => generated 302 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:26.379211 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f200ba-70d7-46c9-9562-591cd6451f70 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:26.396051 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 134/272] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 12 17:19:26.397189 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31b65460-4eb0-43a2-8e9b-1308358a3c59 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:26.420898 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:26.438647 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:26.512171 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f200ba-70d7-46c9-9562-591cd6451f70 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:26.531126 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31b65460-4eb0-43a2-8e9b-1308358a3c59 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:26.562167 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-352305cd-6d5e-4a9b-9ccb-285c9e480e52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:26.565545 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352305cd-6d5e-4a9b-9ccb-285c9e480e52 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:26.565545 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352305cd-6d5e-4a9b-9ccb-285c9e480e52 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:26.565545 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-352305cd-6d5e-4a9b-9ccb-285c9e480e52 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:26.579896 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f372c4-4e56-48be-8729-fb20314f4ab9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:26.580501 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-352305cd-6d5e-4a9b-9ccb-285c9e480e52 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:26.581054 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f372c4-4e56-48be-8729-fb20314f4ab9 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:26.581533 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f372c4-4e56-48be-8729-fb20314f4ab9 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:26.582919 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f372c4-4e56-48be-8729-fb20314f4ab9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:26.583955 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f372c4-4e56-48be-8729-fb20314f4ab9 None admin] RBAC: Adding query filter params (role.id=0c239663092d459e8c9724963e78da76, scope.project.id=31b9ba63c2994be69d5ff224d61a9c25, user.id=84726da6e6d246e5b9500096bf044a36) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:26.584487 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f372c4-4e56-48be-8729-fb20314f4ab9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:26.586215 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f372c4-4e56-48be-8729-fb20314f4ab9 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:26.590277 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-352305cd-6d5e-4a9b-9ccb-285c9e480e52 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 12 17:19:26.590277 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:26.590277 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:26.590277 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:26.590277 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:26.590277 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:26.590277 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:26.590277 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:26.590277 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:26.590277 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:26.590277 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:26.590277 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:26.590277 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:26.590277 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:26.590277 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:26.590277 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:26.591209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:26.591209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:26.591209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:26.591209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:26.591209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:26.591209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:26.591209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:26.591209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:26.591209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:26.591209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:26.591209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:26.591209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:26.591209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:26.591209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:26.591209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:26.591209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:26.592000 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:26.592000 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:26.592000 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:26.592000 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:26.592000 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:26.592000 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:26.592000 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:26.592000 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:26.592000 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:26.592000 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:26.592000 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 12 17:19:26.592000 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:26.592000 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 139/273] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:26 2022] GET /identity/v3/roles/member => generated 84 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:26.593136 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-352305cd-6d5e-4a9b-9ccb-285c9e480e52 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:26.596824 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 135/274] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:19:26 2022] GET /identity/v3/role_assignments?user.id=84726da6e6d246e5b9500096bf044a36&scope.project.id=31b9ba63c2994be69d5ff224d61a9c25&role.id=0c239663092d459e8c9724963e78da76 => generated 608 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:26.598223 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f372c4-4e56-48be-8729-fb20314f4ab9 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:26.725985 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:26.733475 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:26.739948 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-352305cd-6d5e-4a9b-9ccb-285c9e480e52 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:26.743609 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f372c4-4e56-48be-8729-fb20314f4ab9 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:26.781748 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a890c5f-d385-4019-8828-fd900f0abe21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:26.782960 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a890c5f-d385-4019-8828-fd900f0abe21 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:26.783448 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a890c5f-d385-4019-8828-fd900f0abe21 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:26.785523 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a890c5f-d385-4019-8828-fd900f0abe21 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:26.792511 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a890c5f-d385-4019-8828-fd900f0abe21 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:26.794507 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a890c5f-d385-4019-8828-fd900f0abe21 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:26.796628 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 140/275] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:19:26 2022] GET /identity/v3/domains/default => generated 199 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:26.805348 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a890c5f-d385-4019-8828-fd900f0abe21 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:26.817411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ff4cc2-d972-4d0c-8699-32bfaa05588c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:26.817992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ff4cc2-d972-4d0c-8699-32bfaa05588c None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:26.818283 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ff4cc2-d972-4d0c-8699-32bfaa05588c None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:26.818596 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ff4cc2-d972-4d0c-8699-32bfaa05588c None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:26.819368 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ff4cc2-d972-4d0c-8699-32bfaa05588c None admin] RBAC: Adding query filter params (name=member) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:26.819682 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ff4cc2-d972-4d0c-8699-32bfaa05588c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:26.821067 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ff4cc2-d972-4d0c-8699-32bfaa05588c None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:26.834047 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 136/276] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Jul 12 17:19:26 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:26.842972 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ff4cc2-d972-4d0c-8699-32bfaa05588c None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:26.944891 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a890c5f-d385-4019-8828-fd900f0abe21 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:26.977380 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ff4cc2-d972-4d0c-8699-32bfaa05588c None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:26.986609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c515c5f8-a7f4-4700-b675-93bbd1a5b961 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:26.987289 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c515c5f8-a7f4-4700-b675-93bbd1a5b961 None admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:26.987577 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c515c5f8-a7f4-4700-b675-93bbd1a5b961 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:26.988364 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c515c5f8-a7f4-4700-b675-93bbd1a5b961 None admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:26.989556 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c515c5f8-a7f4-4700-b675-93bbd1a5b961 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:26.993548 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c515c5f8-a7f4-4700-b675-93bbd1a5b961 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:27.014702 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9630100-0507-4462-b267-beb0b60c1c3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:27.015582 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9630100-0507-4462-b267-beb0b60c1c3c None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:27.015880 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9630100-0507-4462-b267-beb0b60c1c3c None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:27.016156 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9630100-0507-4462-b267-beb0b60c1c3c None admin] PATH_INFO: `/v3/users/81d8b032e0f04ce8b09407964c1d8318` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:27.029703 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9630100-0507-4462-b267-beb0b60c1c3c None admin] RBAC: Authorizing `identity:get_user(user_id=81d8b032e0f04ce8b09407964c1d8318)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:27.031202 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9630100-0507-4462-b267-beb0b60c1c3c None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:27.032900 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 137/277] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:19:26 2022] GET /identity/v3/users/81d8b032e0f04ce8b09407964c1d8318 => generated 293 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:27.034116 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c689aec3-d0c8-4b70-9926-0f3aca6bb175 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:27.034408 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c689aec3-d0c8-4b70-9926-0f3aca6bb175 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:27.034991 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c689aec3-d0c8-4b70-9926-0f3aca6bb175 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:27.036365 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 138/278] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:27.038930 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c689aec3-d0c8-4b70-9926-0f3aca6bb175 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:27.056682 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:27.142643 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c689aec3-d0c8-4b70-9926-0f3aca6bb175 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:27.166952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 141/279] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Jul 12 17:19:26 2022] POST /identity/v3/users => generated 293 bytes in 364 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:19:27.168443 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef459a2-8fa3-4760-a7c7-3cc2c1d67c33 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:27.168793 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef459a2-8fa3-4760-a7c7-3cc2c1d67c33 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:27.169085 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef459a2-8fa3-4760-a7c7-3cc2c1d67c33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:27.179813 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb23b405-abb7-4623-9b26-77f23f041066 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:27.180497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb23b405-abb7-4623-9b26-77f23f041066 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:27.180790 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb23b405-abb7-4623-9b26-77f23f041066 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:27.181070 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb23b405-abb7-4623-9b26-77f23f041066 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:27.181930 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb23b405-abb7-4623-9b26-77f23f041066 None admin] RBAC: Adding query filter params (role.id=4c5d52f873f94dc1b10918068376add5, user.id=81d8b032e0f04ce8b09407964c1d8318, scope.system=all) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:27.182252 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb23b405-abb7-4623-9b26-77f23f041066 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:27.183776 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb23b405-abb7-4623-9b26-77f23f041066 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:27.189740 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 139/280] 10.222.0.16 () {56 vars in 1308 bytes} [Tue Jul 12 17:19:27 2022] GET /identity/v3/role_assignments?user.id=81d8b032e0f04ce8b09407964c1d8318&scope.system=all&role.id=4c5d52f873f94dc1b10918068376add5 => generated 226 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:19:27.219486 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-0ef459a2-8fa3-4760-a7c7-3cc2c1d67c33 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:27.220817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-0ef459a2-8fa3-4760-a7c7-3cc2c1d67c33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:27.235835 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:27.318373 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 142/281] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:27.344899 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb23b405-abb7-4623-9b26-77f23f041066 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:27.413573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ec7083-6176-4049-9bf2-f081089bda7c None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:27.413852 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ec7083-6176-4049-9bf2-f081089bda7c None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:27.414246 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ec7083-6176-4049-9bf2-f081089bda7c None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:27.417422 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 143/282] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:27 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:27.423085 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b35902d-811a-4649-8aa6-63f397421c91 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:27.423508 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b35902d-811a-4649-8aa6-63f397421c91 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:27.423899 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b35902d-811a-4649-8aa6-63f397421c91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:27.442998 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:27.447234 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-2b35902d-811a-4649-8aa6-63f397421c91 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:27.448467 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-2b35902d-811a-4649-8aa6-63f397421c91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:27.470606 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb23b405-abb7-4623-9b26-77f23f041066 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:27.521573 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9549355f-ee70-45b5-96e4-a58a3f7ebae4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:27.522799 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9549355f-ee70-45b5-96e4-a58a3f7ebae4 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:27.524288 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9549355f-ee70-45b5-96e4-a58a3f7ebae4 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:27.524288 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9549355f-ee70-45b5-96e4-a58a3f7ebae4 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:27.539081 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9549355f-ee70-45b5-96e4-a58a3f7ebae4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:27.543415 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 144/283] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 12 17:19:27.552864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-9549355f-ee70-45b5-96e4-a58a3f7ebae4 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 12 17:19:27.552864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:27.552864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:27.552864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:27.552864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:27.552864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:27.552864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:27.552864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:27.552864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:27.552864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:27.552864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:27.552864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:27.552864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:27.552864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:27.552864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:27.552864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:27.553798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:27.553798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:27.553798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:27.553798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:27.553798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:27.553798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:27.553798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:27.553798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:27.553798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:27.553798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:27.553798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:27.553798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:27.553798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:27.553798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:27.553798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:27.553798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:27.554657 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:27.554657 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:27.554657 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:27.554657 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:27.554657 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:27.554657 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:27.554657 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:27.554657 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:27.554657 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:27.554657 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:27.554657 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 12 17:19:27.554657 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:19:27.554657 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 140/284] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:27 2022] GET /identity/v3/roles/reader => generated 84 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:27.561885 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b35902d-811a-4649-8aa6-63f397421c91 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:27.576378 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9549355f-ee70-45b5-96e4-a58a3f7ebae4 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:27.670474 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:27.696408 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b35902d-811a-4649-8aa6-63f397421c91 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:27.716003 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9549355f-ee70-45b5-96e4-a58a3f7ebae4 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:27.735740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760ae853-8a3b-44d3-b40d-d453d1be860e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:27.736772 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760ae853-8a3b-44d3-b40d-d453d1be860e None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:27.737255 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760ae853-8a3b-44d3-b40d-d453d1be860e None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:27.737724 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760ae853-8a3b-44d3-b40d-d453d1be860e None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:27.739265 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-760ae853-8a3b-44d3-b40d-d453d1be860e None admin] RBAC: Adding query filter params (name=reader) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:27.739784 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-760ae853-8a3b-44d3-b40d-d453d1be860e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:27.741387 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-760ae853-8a3b-44d3-b40d-d453d1be860e None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:27.752140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 145/285] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Jul 12 17:19:27 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:27.758957 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760ae853-8a3b-44d3-b40d-d453d1be860e None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:27.765614 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f246a14-401f-4d40-acb6-10951c9e4d82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:27.766898 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f246a14-401f-4d40-acb6-10951c9e4d82 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:27.768197 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f246a14-401f-4d40-acb6-10951c9e4d82 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:27.768674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f246a14-401f-4d40-acb6-10951c9e4d82 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:27.773614 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:27.787317 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f246a14-401f-4d40-acb6-10951c9e4d82 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:27.798683 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-7f246a14-401f-4d40-acb6-10951c9e4d82 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 12 17:19:27.798683 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:27.798683 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:27.798683 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:27.798683 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:27.798683 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:27.798683 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:27.798683 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:27.798683 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:27.798683 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:27.798683 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:27.798683 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:27.798683 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:27.798683 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:27.798683 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:27.798683 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:27.799606 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:27.799606 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:27.799606 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:27.799606 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:27.799606 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:27.799606 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:27.799606 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:27.799606 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:27.799606 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:27.799606 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:27.799606 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:27.799606 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:27.799606 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:27.799606 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:27.799606 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:27.799606 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:27.800591 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:27.800591 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:27.800591 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:27.800591 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:27.800591 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:27.800591 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:27.800591 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:27.800591 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:27.800591 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:27.800591 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:27.800591 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 12 17:19:27.800591 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:19:27.800591 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 141/286] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:27 2022] GET /identity/v3/roles/member => generated 84 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:19:27.806244 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f246a14-401f-4d40-acb6-10951c9e4d82 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:27.915406 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760ae853-8a3b-44d3-b40d-d453d1be860e None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:27.936661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f246a14-401f-4d40-acb6-10951c9e4d82 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:27.980298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c9c1aa-55dc-4bd0-87f9-9415e1130005 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:27.980980 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c9c1aa-55dc-4bd0-87f9-9415e1130005 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:27.981266 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c9c1aa-55dc-4bd0-87f9-9415e1130005 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:27.981544 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c9c1aa-55dc-4bd0-87f9-9415e1130005 None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:27.982592 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c9c1aa-55dc-4bd0-87f9-9415e1130005 None admin] RBAC: Adding query filter params (name=member) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:27.982905 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c9c1aa-55dc-4bd0-87f9-9415e1130005 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:27.984420 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c9c1aa-55dc-4bd0-87f9-9415e1130005 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:28.000669 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 142/287] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Jul 12 17:19:27 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:28.006874 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c9c1aa-55dc-4bd0-87f9-9415e1130005 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:28.013872 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc0e829-2e07-4626-9813-8064a9eddb7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:28.014837 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc0e829-2e07-4626-9813-8064a9eddb7d None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:28.015130 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc0e829-2e07-4626-9813-8064a9eddb7d None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:28.015411 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc0e829-2e07-4626-9813-8064a9eddb7d None admin] PATH_INFO: `/v3/users/337d5991159749c8bab7a94ad521c19e` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:28.016673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc0e829-2e07-4626-9813-8064a9eddb7d None admin] RBAC: Authorizing `identity:get_user(user_id=337d5991159749c8bab7a94ad521c19e)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:28.018231 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc0e829-2e07-4626-9813-8064a9eddb7d None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:28.020461 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 146/288] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:19:27 2022] GET /identity/v3/users/337d5991159749c8bab7a94ad521c19e => generated 297 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:28.025696 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc0e829-2e07-4626-9813-8064a9eddb7d None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:28.038721 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:28.069793 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:28.115644 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c9c1aa-55dc-4bd0-87f9-9415e1130005 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:28.149884 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb04d04-e1e1-480b-a900-c4c531c7c356 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:28.150808 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb04d04-e1e1-480b-a900-c4c531c7c356 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:28.151095 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb04d04-e1e1-480b-a900-c4c531c7c356 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:28.151372 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb04d04-e1e1-480b-a900-c4c531c7c356 None admin] PATH_INFO: `/v3/users/5536e3dff3c04843adc4527073dd2e59` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:28.152668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb04d04-e1e1-480b-a900-c4c531c7c356 None admin] RBAC: Authorizing `identity:get_user(user_id=5536e3dff3c04843adc4527073dd2e59)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:28.154189 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb04d04-e1e1-480b-a900-c4c531c7c356 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:28.156234 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 143/289] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:19:28 2022] GET /identity/v3/users/5536e3dff3c04843adc4527073dd2e59 => generated 297 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:28.162276 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb04d04-e1e1-480b-a900-c4c531c7c356 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:28.196813 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc0e829-2e07-4626-9813-8064a9eddb7d None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:28.266213 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb04d04-e1e1-480b-a900-c4c531c7c356 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:28.272485 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aad6c33-3309-4913-85f7-dba5a5b392de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:28.273653 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aad6c33-3309-4913-85f7-dba5a5b392de None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:28.274139 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aad6c33-3309-4913-85f7-dba5a5b392de None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:28.274636 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aad6c33-3309-4913-85f7-dba5a5b392de None admin] PATH_INFO: `/v3/projects/31b9ba63c2994be69d5ff224d61a9c25` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:28.279168 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aad6c33-3309-4913-85f7-dba5a5b392de None admin] RBAC: Authorizing `identity:get_project(project_id=31b9ba63c2994be69d5ff224d61a9c25)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:28.280850 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aad6c33-3309-4913-85f7-dba5a5b392de None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:28.283153 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 147/290] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:28 2022] GET /identity/v3/projects/31b9ba63c2994be69d5ff224d61a9c25 => generated 302 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:28.289700 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aad6c33-3309-4913-85f7-dba5a5b392de None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:28.294822 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:28.303204 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-022d94c5-a137-47cb-9963-389a302a1146 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:28.305571 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022d94c5-a137-47cb-9963-389a302a1146 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:28.305571 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022d94c5-a137-47cb-9963-389a302a1146 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:28.305571 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022d94c5-a137-47cb-9963-389a302a1146 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:28.391103 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-022d94c5-a137-47cb-9963-389a302a1146 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:28.392582 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-022d94c5-a137-47cb-9963-389a302a1146 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:28.462916 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:28.471025 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aad6c33-3309-4913-85f7-dba5a5b392de None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:28.535607 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ce367f-2f88-40b2-a4ab-3f34f8f9da28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:28.536769 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ce367f-2f88-40b2-a4ab-3f34f8f9da28 None admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:28.537200 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ce367f-2f88-40b2-a4ab-3f34f8f9da28 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:28.537648 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ce367f-2f88-40b2-a4ab-3f34f8f9da28 None admin] PATH_INFO: `/v3/projects/31b9ba63c2994be69d5ff224d61a9c25/users/337d5991159749c8bab7a94ad521c19e/roles/6670fba31f504efb9ae95813d9b6d660` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:28.542937 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ce367f-2f88-40b2-a4ab-3f34f8f9da28 None admin] RBAC: Authorizing `identity:create_grant(project_id=31b9ba63c2994be69d5ff224d61a9c25, user_id=337d5991159749c8bab7a94ad521c19e, role_id=6670fba31f504efb9ae95813d9b6d660)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:28.544577 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ce367f-2f88-40b2-a4ab-3f34f8f9da28 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:28.587258 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 144/291] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:19:28 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 425 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 12 17:19:28.595647 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-022d94c5-a137-47cb-9963-389a302a1146 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:28.631790 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 148/292] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Jul 12 17:19:28 2022] PUT /identity/v3/projects/31b9ba63c2994be69d5ff224d61a9c25/users/337d5991159749c8bab7a94ad521c19e/roles/6670fba31f504efb9ae95813d9b6d660 => generated 0 bytes in 342 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:19:28.633284 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:28.743077 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-022d94c5-a137-47cb-9963-389a302a1146 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:28.782605 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e16030-0171-4a45-952e-8bafcb49e462 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:28.783502 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e16030-0171-4a45-952e-8bafcb49e462 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:28.783787 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e16030-0171-4a45-952e-8bafcb49e462 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:28.784063 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e16030-0171-4a45-952e-8bafcb49e462 None admin] PATH_INFO: `/v3/projects/31b9ba63c2994be69d5ff224d61a9c25` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:28.785996 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e16030-0171-4a45-952e-8bafcb49e462 None admin] RBAC: Authorizing `identity:get_project(project_id=31b9ba63c2994be69d5ff224d61a9c25)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:28.787600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e16030-0171-4a45-952e-8bafcb49e462 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:28.789881 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 145/293] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:28 2022] GET /identity/v3/projects/31b9ba63c2994be69d5ff224d61a9c25 => generated 302 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:28.797127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ce367f-2f88-40b2-a4ab-3f34f8f9da28 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:28.908896 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ce367f-2f88-40b2-a4ab-3f34f8f9da28 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:28.945533 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d736bf-d15c-41fe-bda5-0cfbb6f91759 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:28.946195 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d736bf-d15c-41fe-bda5-0cfbb6f91759 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:28.946514 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d736bf-d15c-41fe-bda5-0cfbb6f91759 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:28.946795 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d736bf-d15c-41fe-bda5-0cfbb6f91759 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:28.947630 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d736bf-d15c-41fe-bda5-0cfbb6f91759 None admin] RBAC: Adding query filter params (role.id=4c5d52f873f94dc1b10918068376add5, scope.project.id=31b9ba63c2994be69d5ff224d61a9c25, user.id=5536e3dff3c04843adc4527073dd2e59) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:28.947942 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d736bf-d15c-41fe-bda5-0cfbb6f91759 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:28.949444 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d736bf-d15c-41fe-bda5-0cfbb6f91759 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:28.951399 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:28.956327 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 149/294] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:19:28 2022] GET /identity/v3/role_assignments?user.id=5536e3dff3c04843adc4527073dd2e59&scope.project.id=31b9ba63c2994be69d5ff224d61a9c25&role.id=4c5d52f873f94dc1b10918068376add5 => generated 608 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:29.674997 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b87e4e9-8308-4c28-b1d2-741e0a94b6bc None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:29.676225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b87e4e9-8308-4c28-b1d2-741e0a94b6bc None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:29.676225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b87e4e9-8308-4c28-b1d2-741e0a94b6bc None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:29.677717 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 146/295] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:29.683184 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4a9e78-d263-4a9e-8072-b3618dcf94aa None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:29.683588 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4a9e78-d263-4a9e-8072-b3618dcf94aa None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:29.683755 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4a9e78-d263-4a9e-8072-b3618dcf94aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:29.701715 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-df4a9e78-d263-4a9e-8072-b3618dcf94aa None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:29.702961 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-df4a9e78-d263-4a9e-8072-b3618dcf94aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:29.775093 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 150/296] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:29.797452 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b87e4e9-8308-4c28-b1d2-741e0a94b6bc None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:29.908986 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:29.917104 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b87e4e9-8308-4c28-b1d2-741e0a94b6bc None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:29.933270 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cc112c-d34c-4aeb-af43-4ff3afaafb21 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:29.933270 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cc112c-d34c-4aeb-af43-4ff3afaafb21 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:29.933270 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cc112c-d34c-4aeb-af43-4ff3afaafb21 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:29.933270 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 151/297] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:29.933270 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53353d86-2171-47df-9332-65edaedaa1ae None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:29.933270 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53353d86-2171-47df-9332-65edaedaa1ae None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:29.933736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53353d86-2171-47df-9332-65edaedaa1ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:29.950802 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-53353d86-2171-47df-9332-65edaedaa1ae None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:29.952266 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-53353d86-2171-47df-9332-65edaedaa1ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:29.953621 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8839003-d90a-420e-95fd-fa51bf1ecfeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:29.954461 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8839003-d90a-420e-95fd-fa51bf1ecfeb None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:29.954729 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8839003-d90a-420e-95fd-fa51bf1ecfeb None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:29.955003 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8839003-d90a-420e-95fd-fa51bf1ecfeb None admin] PATH_INFO: `/v3/roles/member` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:29.966458 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8839003-d90a-420e-95fd-fa51bf1ecfeb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:29.974896 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-b8839003-d90a-420e-95fd-fa51bf1ecfeb None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 12 17:19:29.974896 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:29.974896 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:29.974896 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:29.974896 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:29.974896 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:29.974896 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:29.974896 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:29.974896 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:29.974896 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:29.974896 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:29.974896 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:29.974896 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:29.974896 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:29.974896 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:29.974896 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:29.975869 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:29.975869 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:29.975869 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:29.975869 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:29.975869 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:29.975869 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:29.975869 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:29.975869 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:29.975869 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:29.975869 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:29.975869 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:29.975869 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:29.975869 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:29.975869 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:29.975869 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:29.975869 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:29.979902 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:29.979902 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:29.979902 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:29.979902 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:29.979902 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:29.979902 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:29.979902 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:29.979902 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:29.979902 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:29.979902 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:29.979902 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 12 17:19:29.979902 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:19:29.979902 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 147/298] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:29 2022] GET /identity/v3/roles/member => generated 84 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:29.979902 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a902fc47-52e4-4471-ae18-2a5603a5ed9c None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:29.980853 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a902fc47-52e4-4471-ae18-2a5603a5ed9c None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:29.980853 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a902fc47-52e4-4471-ae18-2a5603a5ed9c None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:29.980853 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 148/299] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:29 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:29.982345 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a902fc47-52e4-4471-ae18-2a5603a5ed9c None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:30.029877 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 152/300] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:30.031201 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5084f49c-c627-4a29-ad17-739fd7489a4d None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:30.031500 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5084f49c-c627-4a29-ad17-739fd7489a4d None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:30.031785 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5084f49c-c627-4a29-ad17-739fd7489a4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:30.049652 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-5084f49c-c627-4a29-ad17-739fd7489a4d None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:30.051480 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-5084f49c-c627-4a29-ad17-739fd7489a4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:30.102044 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a902fc47-52e4-4471-ae18-2a5603a5ed9c None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:30.110654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:30.126175 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 153/301] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:30.131424 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5084f49c-c627-4a29-ad17-739fd7489a4d None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:30.139915 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06038ded-8b4c-4bb2-b020-93206717b7eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:30.140847 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06038ded-8b4c-4bb2-b020-93206717b7eb None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:30.141375 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06038ded-8b4c-4bb2-b020-93206717b7eb None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:30.141850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06038ded-8b4c-4bb2-b020-93206717b7eb None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:30.142798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06038ded-8b4c-4bb2-b020-93206717b7eb None admin] RBAC: Adding query filter params (name=member) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:30.144875 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06038ded-8b4c-4bb2-b020-93206717b7eb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:30.145698 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06038ded-8b4c-4bb2-b020-93206717b7eb None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:30.154513 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 149/302] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Jul 12 17:19:29 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:30.155588 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06038ded-8b4c-4bb2-b020-93206717b7eb None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:30.176574 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:30.251147 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5084f49c-c627-4a29-ad17-739fd7489a4d None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:30.262164 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06038ded-8b4c-4bb2-b020-93206717b7eb None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:30.288276 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184e19a9-c1bb-45a0-b69a-282ebaf197d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:30.289185 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184e19a9-c1bb-45a0-b69a-282ebaf197d9 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:30.289466 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184e19a9-c1bb-45a0-b69a-282ebaf197d9 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:30.289850 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184e19a9-c1bb-45a0-b69a-282ebaf197d9 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:30.297471 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33f4414-e3bb-483e-99a2-91b66496e869 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:30.298266 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33f4414-e3bb-483e-99a2-91b66496e869 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:30.298591 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33f4414-e3bb-483e-99a2-91b66496e869 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:30.298881 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33f4414-e3bb-483e-99a2-91b66496e869 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:30.300079 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33f4414-e3bb-483e-99a2-91b66496e869 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:30.300502 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-184e19a9-c1bb-45a0-b69a-282ebaf197d9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:30.301543 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33f4414-e3bb-483e-99a2-91b66496e869 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:30.303209 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 150/303] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:19:30 2022] GET /identity/v3/domains/default => generated 199 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:30.303977 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33f4414-e3bb-483e-99a2-91b66496e869 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:30.313014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-184e19a9-c1bb-45a0-b69a-282ebaf197d9 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 12 17:19:30.313014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:30.313014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:30.313014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:30.313014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:30.313014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:30.313014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:30.313014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:30.313014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:30.313014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:30.313014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:30.313014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:30.313014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:30.313014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:30.313014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:30.313014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:30.314103 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:30.314103 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:30.314103 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:30.314103 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:30.314103 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:30.314103 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:30.314103 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:30.314103 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:30.314103 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:30.314103 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:30.314103 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:30.314103 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:30.314103 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:30.314103 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:30.314103 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:30.314103 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:30.315205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:30.315205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:30.315205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:30.315205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:30.315205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:30.315205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:30.315205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:30.315205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:30.315205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:30.315205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:30.315205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 12 17:19:30.315205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:30.315205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 154/304] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:30 2022] GET /identity/v3/roles/reader => generated 84 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:30.315205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184e19a9-c1bb-45a0-b69a-282ebaf197d9 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:30.372147 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:30.404075 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:30.420617 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33f4414-e3bb-483e-99a2-91b66496e869 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:30.429277 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184e19a9-c1bb-45a0-b69a-282ebaf197d9 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:30.458067 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c408913f-dc39-4334-b7eb-089d9c52258a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:30.459060 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c408913f-dc39-4334-b7eb-089d9c52258a None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:30.459350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c408913f-dc39-4334-b7eb-089d9c52258a None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:30.459777 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c408913f-dc39-4334-b7eb-089d9c52258a None admin] PATH_INFO: `/v3/users/81d8b032e0f04ce8b09407964c1d8318` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:30.460851 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c408913f-dc39-4334-b7eb-089d9c52258a None admin] RBAC: Authorizing `identity:get_user(user_id=81d8b032e0f04ce8b09407964c1d8318)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:30.462373 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c408913f-dc39-4334-b7eb-089d9c52258a None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:30.468613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 151/305] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:19:30 2022] GET /identity/v3/users/81d8b032e0f04ce8b09407964c1d8318 => generated 293 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:30.468613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c408913f-dc39-4334-b7eb-089d9c52258a None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:30.470304 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f600f6b-018e-4f6c-b42f-b4381aa68c97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:30.471023 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f600f6b-018e-4f6c-b42f-b4381aa68c97 None admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:30.471310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f600f6b-018e-4f6c-b42f-b4381aa68c97 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:30.471589 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f600f6b-018e-4f6c-b42f-b4381aa68c97 None admin] PATH_INFO: `/v3/groups` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:30.472284 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f600f6b-018e-4f6c-b42f-b4381aa68c97 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:30.473714 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f600f6b-018e-4f6c-b42f-b4381aa68c97 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:30.549543 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 155/306] 10.222.0.16 () {60 vars in 1134 bytes} [Tue Jul 12 17:19:30 2022] POST /identity/v3/groups => generated 229 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:19:30.551442 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f600f6b-018e-4f6c-b42f-b4381aa68c97 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:30.574473 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c408913f-dc39-4334-b7eb-089d9c52258a None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:30.618818 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f02b4b8-8e28-402c-8dac-745e45652f02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:30.619549 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f02b4b8-8e28-402c-8dac-745e45652f02 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:30.619892 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f02b4b8-8e28-402c-8dac-745e45652f02 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:30.620185 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f02b4b8-8e28-402c-8dac-745e45652f02 None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:30.620971 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f02b4b8-8e28-402c-8dac-745e45652f02 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:30.621293 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f02b4b8-8e28-402c-8dac-745e45652f02 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:30.622778 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f02b4b8-8e28-402c-8dac-745e45652f02 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:30.634182 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 152/307] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Jul 12 17:19:30 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:30.643303 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f02b4b8-8e28-402c-8dac-745e45652f02 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:30.669046 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:30.719574 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f600f6b-018e-4f6c-b42f-b4381aa68c97 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:30.732118 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:30.768478 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2420291d-8ab8-4d02-b338-41d19b9fcb66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:30.770070 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2420291d-8ab8-4d02-b338-41d19b9fcb66 None admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:30.770366 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2420291d-8ab8-4d02-b338-41d19b9fcb66 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:30.770723 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2420291d-8ab8-4d02-b338-41d19b9fcb66 None admin] PATH_INFO: `/v3/system/users/81d8b032e0f04ce8b09407964c1d8318/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:30.772300 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2420291d-8ab8-4d02-b338-41d19b9fcb66 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=81d8b032e0f04ce8b09407964c1d8318, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:30.773696 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2420291d-8ab8-4d02-b338-41d19b9fcb66 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:30.788221 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f02b4b8-8e28-402c-8dac-745e45652f02 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:30.827994 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2ad6c2-e05f-455f-8fe1-6355f4ebba65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:30.828877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2ad6c2-e05f-455f-8fe1-6355f4ebba65 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:30.829165 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2ad6c2-e05f-455f-8fe1-6355f4ebba65 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:30.829447 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2ad6c2-e05f-455f-8fe1-6355f4ebba65 None admin] PATH_INFO: `/v3/users/974cba9e4b0b409d8530753bfe2f6b22` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:30.844026 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b2ad6c2-e05f-455f-8fe1-6355f4ebba65 None admin] RBAC: Authorizing `identity:get_user(user_id=974cba9e4b0b409d8530753bfe2f6b22)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:30.848507 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b2ad6c2-e05f-455f-8fe1-6355f4ebba65 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:30.848507 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 153/308] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:19:30 2022] GET /identity/v3/users/974cba9e4b0b409d8530753bfe2f6b22 => generated 293 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:30.848836 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b597c505-0545-45ba-a5d5-f6220d7350f6 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:30.849123 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b597c505-0545-45ba-a5d5-f6220d7350f6 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:30.849409 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b597c505-0545-45ba-a5d5-f6220d7350f6 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:30.850817 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 154/309] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:30.853274 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b597c505-0545-45ba-a5d5-f6220d7350f6 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:30.871728 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 156/310] 10.222.0.16 () {58 vars in 1334 bytes} [Tue Jul 12 17:19:30 2022] PUT /identity/v3/system/users/81d8b032e0f04ce8b09407964c1d8318/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 321 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:19:30.872950 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5782620-5733-43f7-8770-a15f403ecc49 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:30.873250 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5782620-5733-43f7-8770-a15f403ecc49 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:30.873538 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5782620-5733-43f7-8770-a15f403ecc49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:30.892302 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-c5782620-5733-43f7-8770-a15f403ecc49 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:30.893583 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-c5782620-5733-43f7-8770-a15f403ecc49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:30.948519 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:30.984796 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b597c505-0545-45ba-a5d5-f6220d7350f6 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:30.990293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 157/311] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:31.015396 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5782620-5733-43f7-8770-a15f403ecc49 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:31.033698 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51ebd57-ea0b-4517-8f4f-66e96bc560ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:31.035171 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51ebd57-ea0b-4517-8f4f-66e96bc560ce None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:31.035936 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51ebd57-ea0b-4517-8f4f-66e96bc560ce None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:31.036404 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51ebd57-ea0b-4517-8f4f-66e96bc560ce None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:31.037358 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b51ebd57-ea0b-4517-8f4f-66e96bc560ce None admin] RBAC: Adding query filter params (role.id=6670fba31f504efb9ae95813d9b6d660, user.id=974cba9e4b0b409d8530753bfe2f6b22, scope.system=all) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:31.037837 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b51ebd57-ea0b-4517-8f4f-66e96bc560ce None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:31.039565 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b51ebd57-ea0b-4517-8f4f-66e96bc560ce None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:31.046898 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 155/312] 10.222.0.16 () {56 vars in 1308 bytes} [Tue Jul 12 17:19:30 2022] GET /identity/v3/role_assignments?user.id=974cba9e4b0b409d8530753bfe2f6b22&scope.system=all&role.id=6670fba31f504efb9ae95813d9b6d660 => generated 226 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:19:31.148403 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5782620-5733-43f7-8770-a15f403ecc49 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:31.150577 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:31.193907 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b32ae9-c1e9-4f64-a087-77e8ff08469e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:31.195049 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b32ae9-c1e9-4f64-a087-77e8ff08469e None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:31.195500 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b32ae9-c1e9-4f64-a087-77e8ff08469e None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:31.195932 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b32ae9-c1e9-4f64-a087-77e8ff08469e None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:31.208917 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b32ae9-c1e9-4f64-a087-77e8ff08469e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:31.217218 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-06b32ae9-c1e9-4f64-a087-77e8ff08469e None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 12 17:19:31.217218 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:31.217218 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:31.217218 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:31.217218 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:31.217218 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:31.217218 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:31.217218 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:31.217218 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:31.217218 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:31.217218 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:31.217218 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:31.217218 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:31.217218 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:31.217218 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:31.217218 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:31.218408 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:31.218408 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:31.218408 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:31.218408 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:31.218408 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:31.218408 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:31.218408 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:31.218408 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:31.218408 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:31.218408 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:31.218408 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:31.218408 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:31.218408 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:31.218408 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:31.218408 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:31.218408 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:31.219862 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:31.219862 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:31.219862 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:31.219862 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:31.219862 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:31.219862 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:31.219862 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:31.219862 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:31.219862 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:31.219862 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:31.219862 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 12 17:19:31.219862 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:31.219862 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 158/313] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:31 2022] GET /identity/v3/roles/reader => generated 84 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:19:31.225220 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51ebd57-ea0b-4517-8f4f-66e96bc560ce None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:31.336155 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51ebd57-ea0b-4517-8f4f-66e96bc560ce None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:31.374507 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d528ad2-9888-4e62-b289-2960205d7993 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:31.375360 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d528ad2-9888-4e62-b289-2960205d7993 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:31.375817 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d528ad2-9888-4e62-b289-2960205d7993 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:31.376195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d528ad2-9888-4e62-b289-2960205d7993 None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:31.377120 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d528ad2-9888-4e62-b289-2960205d7993 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:31.377533 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d528ad2-9888-4e62-b289-2960205d7993 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:31.379613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d528ad2-9888-4e62-b289-2960205d7993 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:31.382215 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:31.389798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 156/314] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Jul 12 17:19:31 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:31.395589 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b32ae9-c1e9-4f64-a087-77e8ff08469e None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:31.511075 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b32ae9-c1e9-4f64-a087-77e8ff08469e None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:31.546789 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142d5d0e-2999-4fdf-af07-0f0472733ec8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:31.547756 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142d5d0e-2999-4fdf-af07-0f0472733ec8 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:31.548089 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142d5d0e-2999-4fdf-af07-0f0472733ec8 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:31.548389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142d5d0e-2999-4fdf-af07-0f0472733ec8 None admin] PATH_INFO: `/v3/users/337d5991159749c8bab7a94ad521c19e` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:31.549643 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-142d5d0e-2999-4fdf-af07-0f0472733ec8 None admin] RBAC: Authorizing `identity:get_user(user_id=337d5991159749c8bab7a94ad521c19e)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:31.551792 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-142d5d0e-2999-4fdf-af07-0f0472733ec8 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:31.553621 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 159/315] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:19:31 2022] GET /identity/v3/users/337d5991159749c8bab7a94ad521c19e => generated 297 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:31.559173 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d528ad2-9888-4e62-b289-2960205d7993 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:31.669914 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d528ad2-9888-4e62-b289-2960205d7993 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:31.705848 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47184f0-f260-45a0-a3e4-fbc6f234f884 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:31.706783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47184f0-f260-45a0-a3e4-fbc6f234f884 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:31.707079 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47184f0-f260-45a0-a3e4-fbc6f234f884 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:31.707366 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47184f0-f260-45a0-a3e4-fbc6f234f884 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:31.780028 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47184f0-f260-45a0-a3e4-fbc6f234f884 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:31.781624 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47184f0-f260-45a0-a3e4-fbc6f234f884 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:31.795180 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:31.888109 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 157/316] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:19:31 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 12 17:19:31.893600 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142d5d0e-2999-4fdf-af07-0f0472733ec8 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:31.927603 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:32.091660 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142d5d0e-2999-4fdf-af07-0f0472733ec8 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:32.130113 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b289868f-5566-4836-bd6f-c929a0351162 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:32.131401 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b289868f-5566-4836-bd6f-c929a0351162 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:32.131894 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b289868f-5566-4836-bd6f-c929a0351162 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:32.132384 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b289868f-5566-4836-bd6f-c929a0351162 None admin] PATH_INFO: `/v3/projects/31b9ba63c2994be69d5ff224d61a9c25` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:32.133920 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b289868f-5566-4836-bd6f-c929a0351162 None admin] RBAC: Authorizing `identity:get_project(project_id=31b9ba63c2994be69d5ff224d61a9c25)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:32.135683 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b289868f-5566-4836-bd6f-c929a0351162 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:32.138118 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 160/317] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:31 2022] GET /identity/v3/projects/31b9ba63c2994be69d5ff224d61a9c25 => generated 302 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:32.144014 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47184f0-f260-45a0-a3e4-fbc6f234f884 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:32.246201 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:32.251059 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47184f0-f260-45a0-a3e4-fbc6f234f884 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:32.288560 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb48b66-82c7-4ee2-851b-00c564dc3fb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:32.289226 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb48b66-82c7-4ee2-851b-00c564dc3fb5 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:32.289514 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb48b66-82c7-4ee2-851b-00c564dc3fb5 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:32.289796 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb48b66-82c7-4ee2-851b-00c564dc3fb5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:32.290668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb48b66-82c7-4ee2-851b-00c564dc3fb5 None admin] RBAC: Adding query filter params (role.id=6670fba31f504efb9ae95813d9b6d660, scope.project.id=31b9ba63c2994be69d5ff224d61a9c25, user.id=337d5991159749c8bab7a94ad521c19e) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:32.290991 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb48b66-82c7-4ee2-851b-00c564dc3fb5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:32.292451 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb48b66-82c7-4ee2-851b-00c564dc3fb5 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:32.302782 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 158/318] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:19:32 2022] GET /identity/v3/role_assignments?user.id=337d5991159749c8bab7a94ad521c19e&scope.project.id=31b9ba63c2994be69d5ff224d61a9c25&role.id=6670fba31f504efb9ae95813d9b6d660 => generated 608 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:32.741638 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4bea18-ae7e-4b8e-b5d9-dbccc48db81a None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:32.742027 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4bea18-ae7e-4b8e-b5d9-dbccc48db81a None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:32.742205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4bea18-ae7e-4b8e-b5d9-dbccc48db81a None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:32.743759 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 161/319] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:32.748821 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f34add-980e-4558-a283-9d29611edbb3 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:32.749120 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f34add-980e-4558-a283-9d29611edbb3 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:32.749406 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f34add-980e-4558-a283-9d29611edbb3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:32.766159 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-a2f34add-980e-4558-a283-9d29611edbb3 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:32.767411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-a2f34add-980e-4558-a283-9d29611edbb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:32.839171 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 159/320] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:32.861561 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4bea18-ae7e-4b8e-b5d9-dbccc48db81a None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:32.966305 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:32.971000 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4bea18-ae7e-4b8e-b5d9-dbccc48db81a None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:33.007946 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef2426d2-cdd1-42f6-a4b8-eb5160ed467f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:33.008747 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2426d2-cdd1-42f6-a4b8-eb5160ed467f None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:33.009038 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2426d2-cdd1-42f6-a4b8-eb5160ed467f None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:33.009323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef2426d2-cdd1-42f6-a4b8-eb5160ed467f None admin] PATH_INFO: `/v3/domains/default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:33.010671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef2426d2-cdd1-42f6-a4b8-eb5160ed467f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:33.016602 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef2426d2-cdd1-42f6-a4b8-eb5160ed467f None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:33.016602 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 162/321] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:19:32 2022] GET /identity/v3/domains/default => generated 199 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:33.019348 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f34add-980e-4558-a283-9d29611edbb3 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:33.081811 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ba3bae-836e-40ca-8d66-84769f60cf26 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:33.082112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ba3bae-836e-40ca-8d66-84769f60cf26 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:33.082376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ba3bae-836e-40ca-8d66-84769f60cf26 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:33.083921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 163/322] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 12 17:19:33.088602 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad21fbd0-9764-4602-8c4d-7a63bb1b162a None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:33.088893 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad21fbd0-9764-4602-8c4d-7a63bb1b162a None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:33.089228 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad21fbd0-9764-4602-8c4d-7a63bb1b162a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:33.105453 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-ad21fbd0-9764-4602-8c4d-7a63bb1b162a None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:33.106938 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-ad21fbd0-9764-4602-8c4d-7a63bb1b162a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:33.117074 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:33.125997 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f34add-980e-4558-a283-9d29611edbb3 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:33.161996 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b464666-56f5-4cbd-9b16-f96d8553503c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:33.162695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b464666-56f5-4cbd-9b16-f96d8553503c None admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:33.162978 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b464666-56f5-4cbd-9b16-f96d8553503c None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:33.163491 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b464666-56f5-4cbd-9b16-f96d8553503c None admin] PATH_INFO: `/v3/groups` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:33.163945 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b464666-56f5-4cbd-9b16-f96d8553503c None admin] RBAC: Authorizing `identity:create_group()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:33.165474 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b464666-56f5-4cbd-9b16-f96d8553503c None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:33.180797 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 164/323] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 12 17:19:33.204282 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad21fbd0-9764-4602-8c4d-7a63bb1b162a None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:33.297114 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:33.303286 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad21fbd0-9764-4602-8c4d-7a63bb1b162a None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:33.330524 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 160/324] 10.222.0.16 () {60 vars in 1134 bytes} [Tue Jul 12 17:19:33 2022] POST /identity/v3/groups => generated 226 bytes in 312 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:19:33.334706 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2d95fe-250b-4829-abd2-9f4f9176830e None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:33.334706 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2d95fe-250b-4829-abd2-9f4f9176830e None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:33.334706 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2d95fe-250b-4829-abd2-9f4f9176830e None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:33.334706 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 161/325] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:33.338714 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a289b33a-1d61-4b8f-ab9a-612c4bb9eceb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:33.339577 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a289b33a-1d61-4b8f-ab9a-612c4bb9eceb None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:33.339798 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a289b33a-1d61-4b8f-ab9a-612c4bb9eceb None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:33.340075 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a289b33a-1d61-4b8f-ab9a-612c4bb9eceb None admin] PATH_INFO: `/v3/roles/member` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:33.340165 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db34e27-6f4a-45f1-a2f4-476d383b823d None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:33.340474 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db34e27-6f4a-45f1-a2f4-476d383b823d None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:33.340863 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db34e27-6f4a-45f1-a2f4-476d383b823d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:33.352999 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a289b33a-1d61-4b8f-ab9a-612c4bb9eceb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:33.359550 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-9db34e27-6f4a-45f1-a2f4-476d383b823d None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:33.360877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-9db34e27-6f4a-45f1-a2f4-476d383b823d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:33.361389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-a289b33a-1d61-4b8f-ab9a-612c4bb9eceb None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 12 17:19:33.361389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:33.361389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:33.361389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:33.361389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:33.361389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:33.361389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:33.361389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:33.361389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:33.361389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:33.361389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:33.361389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:33.361389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:33.361389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:33.361389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:33.361389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:33.362475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:33.362475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:33.362475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:33.362475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:33.362475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:33.362475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:33.362475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:33.362475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:33.362475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:33.362475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:33.362475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:33.362475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:33.362475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:33.362475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:33.362475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:33.362475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:33.364268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:33.364268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:33.364268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:33.364268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:33.364268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:33.364268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:33.364268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:33.364268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:33.364268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:33.364268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:33.364268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 12 17:19:33.364268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:33.364268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 165/326] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:33 2022] GET /identity/v3/roles/member => generated 84 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:19:33.369312 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a289b33a-1d61-4b8f-ab9a-612c4bb9eceb None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:33.450200 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 162/327] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:33.473605 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db34e27-6f4a-45f1-a2f4-476d383b823d None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:33.497371 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a289b33a-1d61-4b8f-ab9a-612c4bb9eceb None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:33.534822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099c1c5d-d6bb-4494-bf42-14ce9902664b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:33.535989 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099c1c5d-d6bb-4494-bf42-14ce9902664b None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:33.536280 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099c1c5d-d6bb-4494-bf42-14ce9902664b None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:33.536561 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099c1c5d-d6bb-4494-bf42-14ce9902664b None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:33.537321 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-099c1c5d-d6bb-4494-bf42-14ce9902664b None admin] RBAC: Adding query filter params (name=member) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:33.537630 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-099c1c5d-d6bb-4494-bf42-14ce9902664b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:33.539048 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-099c1c5d-d6bb-4494-bf42-14ce9902664b None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:33.552195 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 166/328] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Jul 12 17:19:33 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:19:33.559283 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099c1c5d-d6bb-4494-bf42-14ce9902664b None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:33.584517 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:33.587401 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:33.608168 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db34e27-6f4a-45f1-a2f4-476d383b823d None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:33.653972 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc31983-d3f0-4476-ae13-0bcaa22e6004 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:33.654822 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc31983-d3f0-4476-ae13-0bcaa22e6004 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:33.655158 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc31983-d3f0-4476-ae13-0bcaa22e6004 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:33.655442 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc31983-d3f0-4476-ae13-0bcaa22e6004 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:33.669337 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc31983-d3f0-4476-ae13-0bcaa22e6004 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:33.678017 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-bdc31983-d3f0-4476-ae13-0bcaa22e6004 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 12 17:19:33.678017 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:33.678017 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:33.678017 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:33.678017 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:33.678017 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:33.678017 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:33.678017 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:33.678017 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:33.678017 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:33.678017 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:33.678017 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:33.678017 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:33.678017 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:33.678017 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:33.678017 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:33.679078 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:33.679078 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:33.679078 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:33.679078 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:33.679078 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:33.679078 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:33.679078 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:33.679078 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:33.679078 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:33.679078 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:33.679078 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:33.679078 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:33.679078 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:33.679078 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:33.679078 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:33.679078 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:33.679889 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:33.679889 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:33.679889 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:33.679889 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:33.679889 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:33.679889 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:33.679889 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:33.679889 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:33.679889 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:33.679889 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:33.679889 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 12 17:19:33.679889 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:19:33.680502 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 163/329] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:33 2022] GET /identity/v3/roles/reader => generated 84 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:19:33.686229 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc31983-d3f0-4476-ae13-0bcaa22e6004 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:33.690465 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-099c1c5d-d6bb-4494-bf42-14ce9902664b None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:33.741871 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbee41aa-daf2-45db-a124-ccd2886ce76a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:33.741871 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbee41aa-daf2-45db-a124-ccd2886ce76a None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:33.742291 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbee41aa-daf2-45db-a124-ccd2886ce76a None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:33.742485 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbee41aa-daf2-45db-a124-ccd2886ce76a None admin] PATH_INFO: `/v3/users/81d8b032e0f04ce8b09407964c1d8318` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:33.743738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbee41aa-daf2-45db-a124-ccd2886ce76a None admin] RBAC: Authorizing `identity:get_user(user_id=81d8b032e0f04ce8b09407964c1d8318)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:33.745330 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbee41aa-daf2-45db-a124-ccd2886ce76a None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:33.747144 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 167/330] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:19:33 2022] GET /identity/v3/users/81d8b032e0f04ce8b09407964c1d8318 => generated 293 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:19:33.757191 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbee41aa-daf2-45db-a124-ccd2886ce76a None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:33.824335 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc31983-d3f0-4476-ae13-0bcaa22e6004 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:33.869236 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2316d2-2ac7-4d52-aa88-42cbc4d41f23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:33.869844 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2316d2-2ac7-4d52-aa88-42cbc4d41f23 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:33.870143 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2316d2-2ac7-4d52-aa88-42cbc4d41f23 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:33.870373 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2316d2-2ac7-4d52-aa88-42cbc4d41f23 None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:33.871547 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2316d2-2ac7-4d52-aa88-42cbc4d41f23 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:33.871656 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2316d2-2ac7-4d52-aa88-42cbc4d41f23 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:33.873948 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2316d2-2ac7-4d52-aa88-42cbc4d41f23 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:33.884497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 164/331] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Jul 12 17:19:33 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:19:33.889413 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:33.890919 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2316d2-2ac7-4d52-aa88-42cbc4d41f23 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:33.896440 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:33.905831 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbee41aa-daf2-45db-a124-ccd2886ce76a None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:33.952384 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869efa1a-9346-4da6-b3f6-488c4372fa7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:33.954223 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869efa1a-9346-4da6-b3f6-488c4372fa7f None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:33.954583 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869efa1a-9346-4da6-b3f6-488c4372fa7f None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:33.957306 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869efa1a-9346-4da6-b3f6-488c4372fa7f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:33.957306 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-869efa1a-9346-4da6-b3f6-488c4372fa7f None admin] RBAC: Adding query filter params (role.id=4c5d52f873f94dc1b10918068376add5, user.id=81d8b032e0f04ce8b09407964c1d8318, scope.system=all) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:33.957598 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-869efa1a-9346-4da6-b3f6-488c4372fa7f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:33.959344 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-869efa1a-9346-4da6-b3f6-488c4372fa7f None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:33.976334 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 168/332] 10.222.0.16 () {56 vars in 1308 bytes} [Tue Jul 12 17:19:33 2022] GET /identity/v3/role_assignments?user.id=81d8b032e0f04ce8b09407964c1d8318&scope.system=all&role.id=4c5d52f873f94dc1b10918068376add5 => generated 510 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:34.031707 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2316d2-2ac7-4d52-aa88-42cbc4d41f23 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:34.079424 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37bf4c9a-1e47-4ad2-9b67-76a2bee56085 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:34.080840 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37bf4c9a-1e47-4ad2-9b67-76a2bee56085 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:34.081636 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37bf4c9a-1e47-4ad2-9b67-76a2bee56085 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:34.082473 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37bf4c9a-1e47-4ad2-9b67-76a2bee56085 None admin] PATH_INFO: `/v3/users/974cba9e4b0b409d8530753bfe2f6b22` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:34.084048 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37bf4c9a-1e47-4ad2-9b67-76a2bee56085 None admin] RBAC: Authorizing `identity:get_user(user_id=974cba9e4b0b409d8530753bfe2f6b22)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:34.085742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37bf4c9a-1e47-4ad2-9b67-76a2bee56085 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:34.087780 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 165/333] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:19:33 2022] GET /identity/v3/users/974cba9e4b0b409d8530753bfe2f6b22 => generated 293 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:34.093950 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869efa1a-9346-4da6-b3f6-488c4372fa7f None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:34.223678 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869efa1a-9346-4da6-b3f6-488c4372fa7f None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:34.262728 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ed6aff1-29f3-4b85-85a7-dca5b3c4b00d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:34.263941 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed6aff1-29f3-4b85-85a7-dca5b3c4b00d None admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:34.264424 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed6aff1-29f3-4b85-85a7-dca5b3c4b00d None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:34.264998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed6aff1-29f3-4b85-85a7-dca5b3c4b00d None admin] PATH_INFO: `/v3/system/users/974cba9e4b0b409d8530753bfe2f6b22/roles/6670fba31f504efb9ae95813d9b6d660` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:34.266696 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ed6aff1-29f3-4b85-85a7-dca5b3c4b00d None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=974cba9e4b0b409d8530753bfe2f6b22, role_id=6670fba31f504efb9ae95813d9b6d660)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:34.268323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ed6aff1-29f3-4b85-85a7-dca5b3c4b00d None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:34.395219 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 169/334] 10.222.0.16 () {58 vars in 1334 bytes} [Tue Jul 12 17:19:34 2022] PUT /identity/v3/system/users/974cba9e4b0b409d8530753bfe2f6b22/roles/6670fba31f504efb9ae95813d9b6d660 => generated 0 bytes in 302 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:19:35.736843 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44aacd51-d542-47f8-adec-affbef933ba7 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:35.738112 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44aacd51-d542-47f8-adec-affbef933ba7 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:35.738112 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44aacd51-d542-47f8-adec-affbef933ba7 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:35.740080 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 166/335] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:35.745428 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a614c6-dd60-4bf5-8340-d9728610a30e None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:35.745882 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a614c6-dd60-4bf5-8340-d9728610a30e None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:35.746078 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a614c6-dd60-4bf5-8340-d9728610a30e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:35.771466 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-70a614c6-dd60-4bf5-8340-d9728610a30e None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:35.772680 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-70a614c6-dd60-4bf5-8340-d9728610a30e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:35.775050 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:35.853454 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 170/336] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 12 17:19:35.893345 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44aacd51-d542-47f8-adec-affbef933ba7 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:35.995797 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:36.033464 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44aacd51-d542-47f8-adec-affbef933ba7 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:36.072708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5827b47b-9672-44bc-921f-1fb6c6ce62c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:36.073507 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5827b47b-9672-44bc-921f-1fb6c6ce62c3 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:36.073793 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5827b47b-9672-44bc-921f-1fb6c6ce62c3 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:36.074069 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5827b47b-9672-44bc-921f-1fb6c6ce62c3 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:36.089406 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5827b47b-9672-44bc-921f-1fb6c6ce62c3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:36.097523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-5827b47b-9672-44bc-921f-1fb6c6ce62c3 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 12 17:19:36.097523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:36.097523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:36.097523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:36.097523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:36.097523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:36.097523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:36.097523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:36.097523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:36.097523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:36.097523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:36.097523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:36.097523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:36.097523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:36.097523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:36.097523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:36.098411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:36.098411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:36.098411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:36.098411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:36.098411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:36.098411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:36.098411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:36.098411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:36.098411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:36.098411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:36.098411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:36.098411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:36.098411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:36.098411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:36.098411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:36.098411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:36.099223 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:36.099223 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:36.099223 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:36.099223 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:36.099223 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:36.099223 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:36.099223 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:36.099223 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:36.099223 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:36.099223 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:36.099223 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 12 17:19:36.099223 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:19:36.099223 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 167/337] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:35 2022] GET /identity/v3/roles/member => generated 84 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:36.104694 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a614c6-dd60-4bf5-8340-d9728610a30e None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:36.282539 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a614c6-dd60-4bf5-8340-d9728610a30e None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:36.325846 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb450bbd-c05e-4c19-8710-1d5235515294 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:36.326547 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb450bbd-c05e-4c19-8710-1d5235515294 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:36.326843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb450bbd-c05e-4c19-8710-1d5235515294 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:36.327126 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb450bbd-c05e-4c19-8710-1d5235515294 None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:36.327886 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb450bbd-c05e-4c19-8710-1d5235515294 None admin] RBAC: Adding query filter params (name=member) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:36.328199 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb450bbd-c05e-4c19-8710-1d5235515294 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:36.329573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb450bbd-c05e-4c19-8710-1d5235515294 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:36.339447 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 171/338] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Jul 12 17:19:36 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:36.339756 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:36.346497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5827b47b-9672-44bc-921f-1fb6c6ce62c3 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:36.385038 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafcb068-a0d0-49fc-b234-9824578e71c5 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:36.385207 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafcb068-a0d0-49fc-b234-9824578e71c5 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:36.385444 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafcb068-a0d0-49fc-b234-9824578e71c5 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:36.387119 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 172/339] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:36.391822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3494bdc8-1ac1-4fe3-ac4d-0a4fb2039472 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:36.392113 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3494bdc8-1ac1-4fe3-ac4d-0a4fb2039472 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:36.392399 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3494bdc8-1ac1-4fe3-ac4d-0a4fb2039472 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:36.424054 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-3494bdc8-1ac1-4fe3-ac4d-0a4fb2039472 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:36.425309 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-3494bdc8-1ac1-4fe3-ac4d-0a4fb2039472 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:36.492319 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5827b47b-9672-44bc-921f-1fb6c6ce62c3 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:36.512662 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 173/340] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 12 17:19:36.538476 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:36.547420 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f404fe9-fa5d-4793-97c3-5c77e9171fa6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:36.548405 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f404fe9-fa5d-4793-97c3-5c77e9171fa6 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:36.549004 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f404fe9-fa5d-4793-97c3-5c77e9171fa6 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:36.549543 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f404fe9-fa5d-4793-97c3-5c77e9171fa6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:36.552146 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3494bdc8-1ac1-4fe3-ac4d-0a4fb2039472 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:36.650419 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f404fe9-fa5d-4793-97c3-5c77e9171fa6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:36.651862 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f404fe9-fa5d-4793-97c3-5c77e9171fa6 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:36.729746 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3494bdc8-1ac1-4fe3-ac4d-0a4fb2039472 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:36.785956 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:36.798371 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9dcbf0c-f8e9-44fd-8e0a-1dcc242f5177 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:36.800073 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9dcbf0c-f8e9-44fd-8e0a-1dcc242f5177 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:36.800073 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9dcbf0c-f8e9-44fd-8e0a-1dcc242f5177 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:36.800073 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9dcbf0c-f8e9-44fd-8e0a-1dcc242f5177 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:36.825806 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9dcbf0c-f8e9-44fd-8e0a-1dcc242f5177 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:36.842738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-c9dcbf0c-f8e9-44fd-8e0a-1dcc242f5177 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 12 17:19:36.842738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:36.842738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:36.842738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:36.842738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:36.842738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:36.842738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:36.842738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:36.842738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:36.842738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:36.842738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:36.842738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:36.842738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:36.842738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:36.842738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:36.842738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:36.843685 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:36.843685 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:36.843685 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:36.843685 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:36.843685 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:36.843685 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:36.843685 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:36.843685 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:36.843685 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:36.843685 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:36.843685 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:36.843685 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:36.843685 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:36.843685 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:36.843685 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:36.843685 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:36.844505 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:36.844505 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:36.844505 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:36.844505 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:36.844505 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:36.844505 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:36.844505 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:36.844505 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:36.844505 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:36.844505 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:36.844505 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 12 17:19:36.844505 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:36.844505 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 174/341] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Jul 12 17:19:36 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 293 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:36.847869 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 168/342] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:19:36 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 503 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 12 17:19:36.853871 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9dcbf0c-f8e9-44fd-8e0a-1dcc242f5177 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:36.858451 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f404fe9-fa5d-4793-97c3-5c77e9171fa6 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:36.919523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:36.992887 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9dcbf0c-f8e9-44fd-8e0a-1dcc242f5177 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:37.032042 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2837bec4-7231-45a1-ba6f-d89a3f006f59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:37.032911 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2837bec4-7231-45a1-ba6f-d89a3f006f59 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:37.033193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2837bec4-7231-45a1-ba6f-d89a3f006f59 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:37.033464 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2837bec4-7231-45a1-ba6f-d89a3f006f59 None admin] PATH_INFO: `/v3/projects/4857259d2a364dff960547dc0aac75e0` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:37.036040 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2837bec4-7231-45a1-ba6f-d89a3f006f59 None admin] RBAC: Authorizing `identity:get_project(project_id=4857259d2a364dff960547dc0aac75e0)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:37.037497 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2837bec4-7231-45a1-ba6f-d89a3f006f59 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:37.039252 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 175/343] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:36 2022] GET /identity/v3/projects/4857259d2a364dff960547dc0aac75e0 => generated 298 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:37.045967 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2837bec4-7231-45a1-ba6f-d89a3f006f59 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:37.078072 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f404fe9-fa5d-4793-97c3-5c77e9171fa6 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:37.150650 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f1bb3a-3964-4fff-8f7f-304c58fc5bd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:37.151327 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f1bb3a-3964-4fff-8f7f-304c58fc5bd4 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:37.151611 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f1bb3a-3964-4fff-8f7f-304c58fc5bd4 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:37.151886 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f1bb3a-3964-4fff-8f7f-304c58fc5bd4 None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:37.152691 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f1bb3a-3964-4fff-8f7f-304c58fc5bd4 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:37.153002 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f1bb3a-3964-4fff-8f7f-304c58fc5bd4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:37.154366 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f1bb3a-3964-4fff-8f7f-304c58fc5bd4 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:37.162840 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:37.191802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 169/344] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:19:36 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 320 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:19:37.191802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f1bb3a-3964-4fff-8f7f-304c58fc5bd4 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:37.230739 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2837bec4-7231-45a1-ba6f-d89a3f006f59 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:37.269691 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e3fbf9-e9a3-466e-b977-1ba1c8f2b64a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:37.270867 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e3fbf9-e9a3-466e-b977-1ba1c8f2b64a None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:37.271154 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e3fbf9-e9a3-466e-b977-1ba1c8f2b64a None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:37.271438 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e3fbf9-e9a3-466e-b977-1ba1c8f2b64a None admin] PATH_INFO: `/v3/groups/92246a55b7fd43469a1df54e34c8099f` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:37.278103 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e3fbf9-e9a3-466e-b977-1ba1c8f2b64a None admin] RBAC: Authorizing `identity:get_group(group_id=92246a55b7fd43469a1df54e34c8099f)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:37.279775 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e3fbf9-e9a3-466e-b977-1ba1c8f2b64a None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:37.281835 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 176/345] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Jul 12 17:19:37 2022] GET /identity/v3/groups/92246a55b7fd43469a1df54e34c8099f => generated 226 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:37.291636 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e3fbf9-e9a3-466e-b977-1ba1c8f2b64a None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:37.362834 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:37.370467 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f1bb3a-3964-4fff-8f7f-304c58fc5bd4 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:37.434496 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f239ba-3a4d-409e-b764-3a550f09dfcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:37.435166 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f239ba-3a4d-409e-b764-3a550f09dfcf None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:37.435449 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f239ba-3a4d-409e-b764-3a550f09dfcf None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:37.435724 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f239ba-3a4d-409e-b764-3a550f09dfcf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:37.477712 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e3fbf9-e9a3-466e-b977-1ba1c8f2b64a None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:37.526778 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:37.544909 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1346b8ff-d662-4e3b-af40-aba6c6da6bd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:37.545584 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1346b8ff-d662-4e3b-af40-aba6c6da6bd9 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:37.545873 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1346b8ff-d662-4e3b-af40-aba6c6da6bd9 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:37.546282 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1346b8ff-d662-4e3b-af40-aba6c6da6bd9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:37.547158 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1346b8ff-d662-4e3b-af40-aba6c6da6bd9 None admin] RBAC: Adding query filter params (group.id=92246a55b7fd43469a1df54e34c8099f, role.id=4c5d52f873f94dc1b10918068376add5, scope.project.id=4857259d2a364dff960547dc0aac75e0) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:37.547481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1346b8ff-d662-4e3b-af40-aba6c6da6bd9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:37.549024 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1346b8ff-d662-4e3b-af40-aba6c6da6bd9 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:37.566208 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 177/346] 10.222.0.16 () {56 vars in 1376 bytes} [Tue Jul 12 17:19:37 2022] GET /identity/v3/role_assignments?group.id=92246a55b7fd43469a1df54e34c8099f&scope.project.id=4857259d2a364dff960547dc0aac75e0&role.id=4c5d52f873f94dc1b10918068376add5 => generated 260 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:37.566208 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be38f19-9094-4db2-bbd1-9799eda4ade4 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:37.568335 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be38f19-9094-4db2-bbd1-9799eda4ade4 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:37.568659 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be38f19-9094-4db2-bbd1-9799eda4ade4 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:37.570330 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 178/347] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:37 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:37.582461 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c217ea-585e-4cc6-ab78-8484751d3327 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:37.582461 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c217ea-585e-4cc6-ab78-8484751d3327 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:37.582461 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c217ea-585e-4cc6-ab78-8484751d3327 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:37.583546 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9f239ba-3a4d-409e-b764-3a550f09dfcf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:37.585042 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9f239ba-3a4d-409e-b764-3a550f09dfcf None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:37.616233 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-12c217ea-585e-4cc6-ab78-8484751d3327 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:37.617488 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-12c217ea-585e-4cc6-ab78-8484751d3327 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:37.758647 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 179/348] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:37.759543 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:37.760698 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b7755a-302c-462c-95ff-9e6c6bfde2d2 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:37.760994 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b7755a-302c-462c-95ff-9e6c6bfde2d2 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:37.761275 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b7755a-302c-462c-95ff-9e6c6bfde2d2 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:37.762787 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 180/349] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:37.779944 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 170/350] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:19:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 596 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 12 17:19:37.782024 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2c46ef-663e-4d61-9340-475f873bd4ff None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:37.782324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2c46ef-663e-4d61-9340-475f873bd4ff None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:37.782632 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2c46ef-663e-4d61-9340-475f873bd4ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:37.787532 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f239ba-3a4d-409e-b764-3a550f09dfcf None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:37.815711 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-ca2c46ef-663e-4d61-9340-475f873bd4ff None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:37.817118 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-ca2c46ef-663e-4d61-9340-475f873bd4ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:37.916731 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f239ba-3a4d-409e-b764-3a550f09dfcf None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:37.955492 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 181/351] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:37.956822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2c46ef-663e-4d61-9340-475f873bd4ff None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:37.959284 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b8bc3c-ac3a-4c28-9fc1-56fb187d3cd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:37.960153 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b8bc3c-ac3a-4c28-9fc1-56fb187d3cd8 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:37.960834 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b8bc3c-ac3a-4c28-9fc1-56fb187d3cd8 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:37.961109 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b8bc3c-ac3a-4c28-9fc1-56fb187d3cd8 None admin] PATH_INFO: `/v3/projects/4857259d2a364dff960547dc0aac75e0` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:37.962326 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b8bc3c-ac3a-4c28-9fc1-56fb187d3cd8 None admin] RBAC: Authorizing `identity:get_project(project_id=4857259d2a364dff960547dc0aac75e0)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:37.963814 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b8bc3c-ac3a-4c28-9fc1-56fb187d3cd8 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:37.965861 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 171/352] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:37 2022] GET /identity/v3/projects/4857259d2a364dff960547dc0aac75e0 => generated 298 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:19:37.970534 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec582da-7a59-448a-aabd-8f6b7513943f None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:37.970534 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec582da-7a59-448a-aabd-8f6b7513943f None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:37.970534 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec582da-7a59-448a-aabd-8f6b7513943f None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:37.970534 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 172/353] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:37.973135 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec582da-7a59-448a-aabd-8f6b7513943f None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:38.029565 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:38.053915 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:38.107543 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec582da-7a59-448a-aabd-8f6b7513943f None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:38.109357 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2c46ef-663e-4d61-9340-475f873bd4ff None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:38.151804 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d222b179-3527-4732-9fee-ff5b0ed8a20b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:38.152826 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d222b179-3527-4732-9fee-ff5b0ed8a20b None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:38.152942 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d222b179-3527-4732-9fee-ff5b0ed8a20b None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:38.153196 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d222b179-3527-4732-9fee-ff5b0ed8a20b None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:38.157573 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10e28bf-9d14-4a40-b89a-a8c7f9298a0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:38.158750 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10e28bf-9d14-4a40-b89a-a8c7f9298a0b None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:38.158750 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10e28bf-9d14-4a40-b89a-a8c7f9298a0b None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:38.159024 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10e28bf-9d14-4a40-b89a-a8c7f9298a0b None admin] PATH_INFO: `/v3/groups/92246a55b7fd43469a1df54e34c8099f` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:38.161287 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e10e28bf-9d14-4a40-b89a-a8c7f9298a0b None admin] RBAC: Authorizing `identity:get_group(group_id=92246a55b7fd43469a1df54e34c8099f)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:38.162774 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e10e28bf-9d14-4a40-b89a-a8c7f9298a0b None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:38.166599 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 173/354] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Jul 12 17:19:37 2022] GET /identity/v3/groups/92246a55b7fd43469a1df54e34c8099f => generated 226 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:38.168499 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec899cc9-bc46-48ac-ad75-dfb638eed927 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:38.168975 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec899cc9-bc46-48ac-ad75-dfb638eed927 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:38.169745 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec899cc9-bc46-48ac-ad75-dfb638eed927 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:38.170110 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d222b179-3527-4732-9fee-ff5b0ed8a20b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:38.179557 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-d222b179-3527-4732-9fee-ff5b0ed8a20b None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 12 17:19:38.179557 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:38.179557 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:38.179557 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:38.179557 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:38.179557 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:38.179557 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:38.179557 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:38.179557 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:38.179557 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:38.179557 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:38.179557 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:38.179557 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:38.179557 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:38.179557 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:38.179557 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:38.180549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:38.180549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:38.180549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:38.180549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:38.180549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:38.180549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:38.180549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:38.180549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:38.180549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:38.180549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:38.180549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:38.180549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:38.180549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:38.180549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:38.180549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:38.180549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:38.181351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:38.181351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:38.181351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:38.181351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:38.181351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:38.181351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:38.181351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:38.181351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:38.181351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:38.181351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:38.181351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 12 17:19:38.181351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:38.181351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 182/355] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Jul 12 17:19:37 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:38.182028 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d222b179-3527-4732-9fee-ff5b0ed8a20b None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:38.188572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-ec899cc9-bc46-48ac-ad75-dfb638eed927 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:38.190077 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-ec899cc9-bc46-48ac-ad75-dfb638eed927 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:38.282151 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 174/356] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:38.285418 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8242238a-e574-42bc-85cf-c180643d72bc None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:38.285418 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8242238a-e574-42bc-85cf-c180643d72bc None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:38.285418 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8242238a-e574-42bc-85cf-c180643d72bc None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:38.286353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 175/357] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:38.287029 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8242238a-e574-42bc-85cf-c180643d72bc None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:38.342531 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d222b179-3527-4732-9fee-ff5b0ed8a20b None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:38.355448 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:38.370986 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:38.399337 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f5c1a9-ac06-4358-a721-bb0c1118a160 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:38.400338 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f5c1a9-ac06-4358-a721-bb0c1118a160 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:38.400632 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f5c1a9-ac06-4358-a721-bb0c1118a160 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:38.400919 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f5c1a9-ac06-4358-a721-bb0c1118a160 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:38.414409 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f5c1a9-ac06-4358-a721-bb0c1118a160 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:38.424440 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-89f5c1a9-ac06-4358-a721-bb0c1118a160 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 12 17:19:38.424440 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:38.424440 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:38.424440 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:38.424440 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:38.424440 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:38.424440 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:38.424440 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:38.424440 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:38.424440 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:38.424440 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:38.424440 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:38.424440 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:38.424440 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:38.424440 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:38.424440 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:38.425419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:38.425419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:38.425419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:38.425419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:38.425419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:38.425419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:38.425419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:38.425419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:38.425419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:38.425419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:38.425419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:38.425419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:38.425419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:38.425419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:38.425419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:38.425419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:38.426740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:38.426740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:38.426740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:38.426740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:38.426740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:38.426740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:38.426740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:38.426740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:38.426740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:38.426740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:38.426740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 12 17:19:38.426740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:38.426740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 183/358] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:38 2022] GET /identity/v3/roles/member => generated 84 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:38.427629 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f5c1a9-ac06-4358-a721-bb0c1118a160 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:38.429085 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8242238a-e574-42bc-85cf-c180643d72bc None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:38.477177 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ac88b9-e020-4a6a-9968-c25f04b86555 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:38.478128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ac88b9-e020-4a6a-9968-c25f04b86555 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:38.478627 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ac88b9-e020-4a6a-9968-c25f04b86555 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:38.479099 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ac88b9-e020-4a6a-9968-c25f04b86555 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:38.480325 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ac88b9-e020-4a6a-9968-c25f04b86555 None admin] RBAC: Adding query filter params (group.id=92246a55b7fd43469a1df54e34c8099f, role.id=0c239663092d459e8c9724963e78da76, scope.project.id=4857259d2a364dff960547dc0aac75e0) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:38.480645 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ac88b9-e020-4a6a-9968-c25f04b86555 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:38.482348 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ac88b9-e020-4a6a-9968-c25f04b86555 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:38.489888 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 176/359] 10.222.0.16 () {56 vars in 1376 bytes} [Tue Jul 12 17:19:38 2022] GET /identity/v3/role_assignments?group.id=92246a55b7fd43469a1df54e34c8099f&scope.project.id=4857259d2a364dff960547dc0aac75e0&role.id=0c239663092d459e8c9724963e78da76 => generated 260 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:38.491518 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4e49eb-3870-40ab-97b7-6dd8685f806d None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:38.491810 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4e49eb-3870-40ab-97b7-6dd8685f806d None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:38.492092 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4e49eb-3870-40ab-97b7-6dd8685f806d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:38.515894 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-3d4e49eb-3870-40ab-97b7-6dd8685f806d None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:38.517136 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-3d4e49eb-3870-40ab-97b7-6dd8685f806d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:38.562196 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f5c1a9-ac06-4358-a721-bb0c1118a160 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:38.606239 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6c7521-4c04-419c-86f5-cd81a5b32cb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:38.606964 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6c7521-4c04-419c-86f5-cd81a5b32cb3 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:38.607253 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6c7521-4c04-419c-86f5-cd81a5b32cb3 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:38.607540 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6c7521-4c04-419c-86f5-cd81a5b32cb3 None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:38.608366 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6c7521-4c04-419c-86f5-cd81a5b32cb3 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:38.608731 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6c7521-4c04-419c-86f5-cd81a5b32cb3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:38.610170 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6c7521-4c04-419c-86f5-cd81a5b32cb3 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:38.620647 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 184/360] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:19:38 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:38.621231 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6c7521-4c04-419c-86f5-cd81a5b32cb3 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:38.635936 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 177/361] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:38.637229 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4e49eb-3870-40ab-97b7-6dd8685f806d None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:38.657589 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:38.716733 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:38.755387 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6c7521-4c04-419c-86f5-cd81a5b32cb3 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:38.779670 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4e49eb-3870-40ab-97b7-6dd8685f806d None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:38.802879 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077733dc-02b2-4406-a083-3190dc622b8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:38.803692 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077733dc-02b2-4406-a083-3190dc622b8d None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:38.803979 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077733dc-02b2-4406-a083-3190dc622b8d None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:38.804258 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077733dc-02b2-4406-a083-3190dc622b8d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:38.816731 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077733dc-02b2-4406-a083-3190dc622b8d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:38.824734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-077733dc-02b2-4406-a083-3190dc622b8d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:19:38.824734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:38.824734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:38.824734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:38.824734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:38.824734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:38.824734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:38.824734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:38.824734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:38.824734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:38.824734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:38.824734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:38.824734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:38.824734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:38.824734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:38.824734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:38.825629 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:38.825629 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:38.825629 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:38.825629 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:38.825629 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:38.825629 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:38.825629 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:38.825629 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:38.825629 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:38.825629 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:38.825629 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:38.825629 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:38.825629 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:38.825629 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:38.825629 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:38.825629 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:38.826880 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:38.826880 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:38.826880 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:38.826880 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:38.826880 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:38.826880 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:38.826880 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:38.826880 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:38.826880 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:38.826880 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:38.826880 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:19:38.826880 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:38.826880 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 185/362] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Jul 12 17:19:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:38.830216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb9b95d-d844-404f-9347-a7319297b8ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:38.831069 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077733dc-02b2-4406-a083-3190dc622b8d None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:38.831455 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb9b95d-d844-404f-9347-a7319297b8ac None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:38.831930 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb9b95d-d844-404f-9347-a7319297b8ac None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:38.832601 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb9b95d-d844-404f-9347-a7319297b8ac None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:38.833585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb9b95d-d844-404f-9347-a7319297b8ac None admin] RBAC: Adding query filter params (name=member) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:38.834107 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb9b95d-d844-404f-9347-a7319297b8ac None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:38.835803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb9b95d-d844-404f-9347-a7319297b8ac None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:38.846778 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 178/363] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Jul 12 17:19:38 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:38.848146 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb9b95d-d844-404f-9347-a7319297b8ac None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:38.930672 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:38.944225 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077733dc-02b2-4406-a083-3190dc622b8d None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:38.966110 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb9b95d-d844-404f-9347-a7319297b8ac None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:38.981240 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c894b1a9-3ed7-42d4-be54-219bc2d94ff7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:38.982013 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c894b1a9-3ed7-42d4-be54-219bc2d94ff7 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:38.982361 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c894b1a9-3ed7-42d4-be54-219bc2d94ff7 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:38.982746 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c894b1a9-3ed7-42d4-be54-219bc2d94ff7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:38.986120 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:39.008161 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb878ac-3271-4504-b957-c06061ea37fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:39.009366 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb878ac-3271-4504-b957-c06061ea37fb None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:39.009863 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb878ac-3271-4504-b957-c06061ea37fb None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:39.010317 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb878ac-3271-4504-b957-c06061ea37fb None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:39.026492 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb878ac-3271-4504-b957-c06061ea37fb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:39.035722 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-1fb878ac-3271-4504-b957-c06061ea37fb None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 12 17:19:39.035722 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:39.035722 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:39.035722 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:39.035722 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:39.035722 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:39.035722 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:39.035722 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:39.035722 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:39.035722 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:39.035722 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:39.035722 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:39.035722 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:39.035722 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:39.035722 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:39.035722 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:39.036857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:39.036857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:39.036857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:39.036857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:39.036857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:39.036857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:39.036857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:39.036857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:39.036857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:39.036857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:39.036857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:39.036857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:39.036857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:39.036857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:39.036857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:39.036857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:39.037692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:39.037692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:39.037692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:39.037692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:39.037692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:39.037692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:39.037692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:39.037692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:39.037692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:39.037692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:39.037692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 12 17:19:39.037692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:19:39.040228 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 179/364] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:38 2022] GET /identity/v3/roles/reader => generated 84 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:39.042384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb878ac-3271-4504-b957-c06061ea37fb None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:39.072474 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c894b1a9-3ed7-42d4-be54-219bc2d94ff7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:39.074272 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c894b1a9-3ed7-42d4-be54-219bc2d94ff7 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:39.193745 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:39.210399 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 186/365] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:19:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 384 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 12 17:19:39.211278 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c894b1a9-3ed7-42d4-be54-219bc2d94ff7 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:39.229334 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb878ac-3271-4504-b957-c06061ea37fb None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:39.285107 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58c51d8-6547-4f9b-867f-f8461cfc03d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:39.286187 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58c51d8-6547-4f9b-867f-f8461cfc03d5 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:39.286502 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58c51d8-6547-4f9b-867f-f8461cfc03d5 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:39.286789 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58c51d8-6547-4f9b-867f-f8461cfc03d5 None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:39.287555 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58c51d8-6547-4f9b-867f-f8461cfc03d5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:39.287871 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58c51d8-6547-4f9b-867f-f8461cfc03d5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:39.290233 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58c51d8-6547-4f9b-867f-f8461cfc03d5 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:39.301992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 180/366] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:39 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:39.303012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58c51d8-6547-4f9b-867f-f8461cfc03d5 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:39.362549 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:39.375897 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c894b1a9-3ed7-42d4-be54-219bc2d94ff7 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:39.415207 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6653a42b-75e8-4ad1-88cf-aa2cc192d4c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:39.415880 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6653a42b-75e8-4ad1-88cf-aa2cc192d4c2 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:39.416171 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6653a42b-75e8-4ad1-88cf-aa2cc192d4c2 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:39.416456 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6653a42b-75e8-4ad1-88cf-aa2cc192d4c2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:39.434547 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58c51d8-6547-4f9b-867f-f8461cfc03d5 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:39.475500 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e4e070-b44f-43f4-8ea7-b7fd5a1221f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:39.476173 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e4e070-b44f-43f4-8ea7-b7fd5a1221f6 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:39.476467 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e4e070-b44f-43f4-8ea7-b7fd5a1221f6 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:39.476749 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51e4e070-b44f-43f4-8ea7-b7fd5a1221f6 None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:39.477618 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51e4e070-b44f-43f4-8ea7-b7fd5a1221f6 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:39.477941 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51e4e070-b44f-43f4-8ea7-b7fd5a1221f6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:39.479363 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51e4e070-b44f-43f4-8ea7-b7fd5a1221f6 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:39.481562 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:39.489552 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 181/367] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Jul 12 17:19:39 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:39.499060 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e4e070-b44f-43f4-8ea7-b7fd5a1221f6 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:39.499957 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6653a42b-75e8-4ad1-88cf-aa2cc192d4c2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:39.501589 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6653a42b-75e8-4ad1-88cf-aa2cc192d4c2 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:39.603384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:39.606625 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51e4e070-b44f-43f4-8ea7-b7fd5a1221f6 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:39.619481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 187/368] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:19:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 408 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 12 17:19:39.620521 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6653a42b-75e8-4ad1-88cf-aa2cc192d4c2 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:39.646794 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30bbf89-06a3-4d10-bb7e-f7e0596fb231 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:39.647692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30bbf89-06a3-4d10-bb7e-f7e0596fb231 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:39.648140 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30bbf89-06a3-4d10-bb7e-f7e0596fb231 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:39.648351 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30bbf89-06a3-4d10-bb7e-f7e0596fb231 None admin] PATH_INFO: `/v3/projects/31b9ba63c2994be69d5ff224d61a9c25` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:39.649579 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b30bbf89-06a3-4d10-bb7e-f7e0596fb231 None admin] RBAC: Authorizing `identity:get_project(project_id=31b9ba63c2994be69d5ff224d61a9c25)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:39.651077 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b30bbf89-06a3-4d10-bb7e-f7e0596fb231 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:39.653224 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 182/369] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:39 2022] GET /identity/v3/projects/31b9ba63c2994be69d5ff224d61a9c25 => generated 302 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:39.654017 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30bbf89-06a3-4d10-bb7e-f7e0596fb231 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:39.728553 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:39.743419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6653a42b-75e8-4ad1-88cf-aa2cc192d4c2 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:39.765950 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30bbf89-06a3-4d10-bb7e-f7e0596fb231 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:39.781624 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3720f27a-3768-4c62-9c4a-d7892bd65090 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:39.782374 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3720f27a-3768-4c62-9c4a-d7892bd65090 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:39.782967 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3720f27a-3768-4c62-9c4a-d7892bd65090 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:39.783126 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3720f27a-3768-4c62-9c4a-d7892bd65090 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:39.805471 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84fb816-51ad-4ea0-ac4b-427232f31831 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:39.806720 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84fb816-51ad-4ea0-ac4b-427232f31831 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:39.807313 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84fb816-51ad-4ea0-ac4b-427232f31831 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:39.807790 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84fb816-51ad-4ea0-ac4b-427232f31831 None admin] PATH_INFO: `/v3/users/974cba9e4b0b409d8530753bfe2f6b22` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:39.809526 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a84fb816-51ad-4ea0-ac4b-427232f31831 None admin] RBAC: Authorizing `identity:get_user(user_id=974cba9e4b0b409d8530753bfe2f6b22)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:39.812361 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a84fb816-51ad-4ea0-ac4b-427232f31831 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:39.816491 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 183/370] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:19:39 2022] GET /identity/v3/users/974cba9e4b0b409d8530753bfe2f6b22 => generated 293 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:39.817545 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84fb816-51ad-4ea0-ac4b-427232f31831 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:39.866068 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:39.878353 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3720f27a-3768-4c62-9c4a-d7892bd65090 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:39.878353 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3720f27a-3768-4c62-9c4a-d7892bd65090 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:39.942975 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84fb816-51ad-4ea0-ac4b-427232f31831 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:39.978990 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:39.981390 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea5aead-cc29-4f2b-a34a-dc828e839bb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:39.982852 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea5aead-cc29-4f2b-a34a-dc828e839bb4 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:39.983337 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea5aead-cc29-4f2b-a34a-dc828e839bb4 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:39.983810 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea5aead-cc29-4f2b-a34a-dc828e839bb4 None admin] PATH_INFO: `/v3/projects/31b9ba63c2994be69d5ff224d61a9c25` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:39.985242 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea5aead-cc29-4f2b-a34a-dc828e839bb4 None admin] RBAC: Authorizing `identity:get_project(project_id=31b9ba63c2994be69d5ff224d61a9c25)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:39.987500 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea5aead-cc29-4f2b-a34a-dc828e839bb4 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:39.990204 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 184/371] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:39 2022] GET /identity/v3/projects/31b9ba63c2994be69d5ff224d61a9c25 => generated 302 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:39.991853 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea5aead-cc29-4f2b-a34a-dc828e839bb4 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:40.001753 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 188/372] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:19:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 12 17:19:40.007214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3720f27a-3768-4c62-9c4a-d7892bd65090 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:40.139852 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3720f27a-3768-4c62-9c4a-d7892bd65090 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:40.144932 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea5aead-cc29-4f2b-a34a-dc828e839bb4 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:40.149663 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:40.182006 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60759655-a3c3-4729-9c26-2823394379a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:40.182006 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60759655-a3c3-4729-9c26-2823394379a0 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:40.182006 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60759655-a3c3-4729-9c26-2823394379a0 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:40.182006 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60759655-a3c3-4729-9c26-2823394379a0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:40.185782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60759655-a3c3-4729-9c26-2823394379a0 None admin] RBAC: Adding query filter params (role.id=6670fba31f504efb9ae95813d9b6d660, user.id=974cba9e4b0b409d8530753bfe2f6b22, scope.system=all) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:40.185782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60759655-a3c3-4729-9c26-2823394379a0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:40.185782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60759655-a3c3-4729-9c26-2823394379a0 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:40.191382 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc1f01b2-2b3e-421f-ab6f-553e3296fe9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:40.192320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1f01b2-2b3e-421f-ab6f-553e3296fe9d None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:40.193275 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1f01b2-2b3e-421f-ab6f-553e3296fe9d None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:40.193956 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1f01b2-2b3e-421f-ab6f-553e3296fe9d None admin] PATH_INFO: `/v3/groups/92246a55b7fd43469a1df54e34c8099f` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:40.196788 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc1f01b2-2b3e-421f-ab6f-553e3296fe9d None admin] RBAC: Authorizing `identity:get_group(group_id=92246a55b7fd43469a1df54e34c8099f)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:40.198310 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc1f01b2-2b3e-421f-ab6f-553e3296fe9d None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:40.200308 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 185/373] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Jul 12 17:19:39 2022] GET /identity/v3/groups/92246a55b7fd43469a1df54e34c8099f => generated 226 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:40.205145 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 189/374] 10.222.0.16 () {56 vars in 1308 bytes} [Tue Jul 12 17:19:40 2022] GET /identity/v3/role_assignments?user.id=974cba9e4b0b409d8530753bfe2f6b22&scope.system=all&role.id=6670fba31f504efb9ae95813d9b6d660 => generated 510 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:40.205312 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc1f01b2-2b3e-421f-ab6f-553e3296fe9d None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:40.205493 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60759655-a3c3-4729-9c26-2823394379a0 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:40.279722 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:40.361619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60759655-a3c3-4729-9c26-2823394379a0 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:40.368237 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc1f01b2-2b3e-421f-ab6f-553e3296fe9d None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:40.413451 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40207724-e9e5-4712-8068-b6eeb9d65d41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:40.414370 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40207724-e9e5-4712-8068-b6eeb9d65d41 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:40.414773 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40207724-e9e5-4712-8068-b6eeb9d65d41 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:40.415159 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40207724-e9e5-4712-8068-b6eeb9d65d41 None admin] PATH_INFO: `/v3/projects/1d1f69915b004b66af4c48b7c59a15d8` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:40.415937 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9365de6-c1d8-41db-99ac-591468b38163 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:40.416493 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40207724-e9e5-4712-8068-b6eeb9d65d41 None admin] RBAC: Authorizing `identity:get_project(project_id=1d1f69915b004b66af4c48b7c59a15d8)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:40.417124 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9365de6-c1d8-41db-99ac-591468b38163 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:40.417548 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9365de6-c1d8-41db-99ac-591468b38163 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:40.417972 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9365de6-c1d8-41db-99ac-591468b38163 None admin] PATH_INFO: `/v3/groups/92246a55b7fd43469a1df54e34c8099f` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:40.418151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40207724-e9e5-4712-8068-b6eeb9d65d41 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:40.419188 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9365de6-c1d8-41db-99ac-591468b38163 None admin] RBAC: Authorizing `identity:get_group(group_id=92246a55b7fd43469a1df54e34c8099f)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:40.420604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9365de6-c1d8-41db-99ac-591468b38163 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:40.422605 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 186/375] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Jul 12 17:19:40 2022] GET /identity/v3/groups/92246a55b7fd43469a1df54e34c8099f => generated 226 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:40.423872 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ee940c-3d8a-4868-9503-507892c24d56 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:40.423981 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 190/376] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:40 2022] GET /identity/v3/projects/1d1f69915b004b66af4c48b7c59a15d8 => generated 344 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:40.424744 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ee940c-3d8a-4868-9503-507892c24d56 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:40.425188 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ee940c-3d8a-4868-9503-507892c24d56 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:40.425306 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40207724-e9e5-4712-8068-b6eeb9d65d41 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:40.427589 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 187/377] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:40.428302 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ee940c-3d8a-4868-9503-507892c24d56 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:40.489182 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:40.563199 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40207724-e9e5-4712-8068-b6eeb9d65d41 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:40.571302 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ee940c-3d8a-4868-9503-507892c24d56 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:40.584782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:40.610591 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a73617c-a83f-4d05-a106-819bc3783bfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:40.611260 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a73617c-a83f-4d05-a106-819bc3783bfa None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:40.612147 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a73617c-a83f-4d05-a106-819bc3783bfa None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:40.612758 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a73617c-a83f-4d05-a106-819bc3783bfa None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:40.613609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a73617c-a83f-4d05-a106-819bc3783bfa None admin] RBAC: Adding query filter params (group.id=92246a55b7fd43469a1df54e34c8099f, role.id=0c239663092d459e8c9724963e78da76, scope.project.id=31b9ba63c2994be69d5ff224d61a9c25) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:40.613931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a73617c-a83f-4d05-a106-819bc3783bfa None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:40.615511 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a73617c-a83f-4d05-a106-819bc3783bfa None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:40.618885 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269e4329-2a6f-4640-83a0-c8504b1df2dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:40.619548 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269e4329-2a6f-4640-83a0-c8504b1df2dc None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:40.619824 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269e4329-2a6f-4640-83a0-c8504b1df2dc None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:40.620577 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269e4329-2a6f-4640-83a0-c8504b1df2dc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:40.621435 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269e4329-2a6f-4640-83a0-c8504b1df2dc None admin] RBAC: Adding query filter params (group.id=92246a55b7fd43469a1df54e34c8099f, role.id=4c5d52f873f94dc1b10918068376add5, scope.project.id=31b9ba63c2994be69d5ff224d61a9c25) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:40.621939 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269e4329-2a6f-4640-83a0-c8504b1df2dc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:40.625991 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269e4329-2a6f-4640-83a0-c8504b1df2dc None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:40.626126 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 191/378] 10.222.0.16 () {56 vars in 1376 bytes} [Tue Jul 12 17:19:40 2022] GET /identity/v3/role_assignments?group.id=92246a55b7fd43469a1df54e34c8099f&scope.project.id=31b9ba63c2994be69d5ff224d61a9c25&role.id=0c239663092d459e8c9724963e78da76 => generated 260 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:40.626126 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a73617c-a83f-4d05-a106-819bc3783bfa None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:40.630850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 188/379] 10.222.0.16 () {56 vars in 1376 bytes} [Tue Jul 12 17:19:40 2022] GET /identity/v3/role_assignments?group.id=92246a55b7fd43469a1df54e34c8099f&scope.project.id=31b9ba63c2994be69d5ff224d61a9c25&role.id=4c5d52f873f94dc1b10918068376add5 => generated 260 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:19:40.632186 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fce8679-464f-40ad-b94d-9e865c7dae63 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:40.632477 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fce8679-464f-40ad-b94d-9e865c7dae63 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:40.632758 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fce8679-464f-40ad-b94d-9e865c7dae63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:40.655058 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-6fce8679-464f-40ad-b94d-9e865c7dae63 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:40.655058 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-6fce8679-464f-40ad-b94d-9e865c7dae63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:40.758108 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 189/380] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:40.782527 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fce8679-464f-40ad-b94d-9e865c7dae63 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:40.843975 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a73617c-a83f-4d05-a106-819bc3783bfa None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:40.867499 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:40.902370 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bff157c-fd1b-482d-8162-eca1bb032a58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:40.904468 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bff157c-fd1b-482d-8162-eca1bb032a58 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:40.905005 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bff157c-fd1b-482d-8162-eca1bb032a58 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:40.905529 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bff157c-fd1b-482d-8162-eca1bb032a58 None admin] PATH_INFO: `/v3/groups/059e96df72874559ab13e17f5c9d5cfc` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:40.912920 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bff157c-fd1b-482d-8162-eca1bb032a58 None admin] RBAC: Authorizing `identity:get_group(group_id=059e96df72874559ab13e17f5c9d5cfc)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:40.915585 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bff157c-fd1b-482d-8162-eca1bb032a58 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:40.918777 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 192/381] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Jul 12 17:19:40 2022] GET /identity/v3/groups/059e96df72874559ab13e17f5c9d5cfc => generated 229 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:40.925314 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bff157c-fd1b-482d-8162-eca1bb032a58 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:40.981786 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:40.991763 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fce8679-464f-40ad-b94d-9e865c7dae63 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:41.045325 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c95c370-9a1a-4255-b8a1-2738f2a46c5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:41.046307 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c95c370-9a1a-4255-b8a1-2738f2a46c5e None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:41.047061 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c95c370-9a1a-4255-b8a1-2738f2a46c5e None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:41.047349 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c95c370-9a1a-4255-b8a1-2738f2a46c5e None admin] PATH_INFO: `/v3/roles/member` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:41.058659 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bff157c-fd1b-482d-8162-eca1bb032a58 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:41.061864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c95c370-9a1a-4255-b8a1-2738f2a46c5e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:41.072265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-3c95c370-9a1a-4255-b8a1-2738f2a46c5e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 12 17:19:41.072265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:41.072265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:41.072265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:41.072265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:41.072265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:41.072265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:41.072265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:41.072265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:41.072265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:41.072265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:41.072265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:41.072265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:41.072265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:41.072265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:41.072265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:41.073216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:41.073216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:41.073216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:41.073216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:41.073216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:41.073216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:41.073216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:41.073216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:41.073216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:41.073216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:41.073216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:41.073216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:41.073216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:41.073216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:41.073216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:41.073216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:41.074028 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:41.074028 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:41.074028 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:41.074028 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:41.074028 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:41.074028 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:41.074028 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:41.074028 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:41.074028 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:41.074028 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:41.074028 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 12 17:19:41.074028 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:19:41.074028 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 190/382] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:40 2022] GET /identity/v3/roles/member => generated 84 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:19:41.078466 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d018cc8-778e-4720-830c-f1995a88c620 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:41.078466 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d018cc8-778e-4720-830c-f1995a88c620 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:41.078466 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d018cc8-778e-4720-830c-f1995a88c620 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:41.080143 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 191/383] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:41.082018 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d018cc8-778e-4720-830c-f1995a88c620 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:41.105025 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9384c26f-76e3-43e8-b71f-5a8455bedd08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:41.106138 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9384c26f-76e3-43e8-b71f-5a8455bedd08 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:41.106552 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9384c26f-76e3-43e8-b71f-5a8455bedd08 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:41.106899 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9384c26f-76e3-43e8-b71f-5a8455bedd08 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:41.107792 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9384c26f-76e3-43e8-b71f-5a8455bedd08 None admin] RBAC: Adding query filter params (group.id=059e96df72874559ab13e17f5c9d5cfc, role.id=eb05bf286c894ea9bf44a6c023a4323b, scope.project.id=1d1f69915b004b66af4c48b7c59a15d8) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:41.108167 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9384c26f-76e3-43e8-b71f-5a8455bedd08 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:41.109703 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9384c26f-76e3-43e8-b71f-5a8455bedd08 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:41.116961 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 193/384] 10.222.0.16 () {56 vars in 1376 bytes} [Tue Jul 12 17:19:40 2022] GET /identity/v3/role_assignments?group.id=059e96df72874559ab13e17f5c9d5cfc&scope.project.id=1d1f69915b004b66af4c48b7c59a15d8&role.id=eb05bf286c894ea9bf44a6c023a4323b => generated 260 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:19:41.118343 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200a1e3a-a02a-42b5-aa71-931bb195b8ab None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:41.118772 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200a1e3a-a02a-42b5-aa71-931bb195b8ab None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:41.119061 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200a1e3a-a02a-42b5-aa71-931bb195b8ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:41.140681 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-200a1e3a-a02a-42b5-aa71-931bb195b8ab None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:41.141947 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-200a1e3a-a02a-42b5-aa71-931bb195b8ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:41.228934 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 194/385] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:41.254004 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200a1e3a-a02a-42b5-aa71-931bb195b8ab None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:41.260859 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:41.280426 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d018cc8-778e-4720-830c-f1995a88c620 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:41.301377 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:41.356020 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5cce2d8-b088-47cc-ae7b-01e1028640eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:41.356701 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cce2d8-b088-47cc-ae7b-01e1028640eb None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:41.356986 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cce2d8-b088-47cc-ae7b-01e1028640eb None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:41.357397 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cce2d8-b088-47cc-ae7b-01e1028640eb None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:41.358175 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5cce2d8-b088-47cc-ae7b-01e1028640eb None admin] RBAC: Adding query filter params (name=member) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:41.358504 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5cce2d8-b088-47cc-ae7b-01e1028640eb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:41.359905 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5cce2d8-b088-47cc-ae7b-01e1028640eb None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:41.377063 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200a1e3a-a02a-42b5-aa71-931bb195b8ab None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:41.378364 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 192/386] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Jul 12 17:19:41 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:19:41.383842 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5cce2d8-b088-47cc-ae7b-01e1028640eb None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:41.427782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d312b0f7-59f6-485e-83f7-a6d5d3de8a82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:41.428611 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d312b0f7-59f6-485e-83f7-a6d5d3de8a82 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:41.428899 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d312b0f7-59f6-485e-83f7-a6d5d3de8a82 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:41.429181 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d312b0f7-59f6-485e-83f7-a6d5d3de8a82 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:41.447254 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d312b0f7-59f6-485e-83f7-a6d5d3de8a82 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:41.463398 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-d312b0f7-59f6-485e-83f7-a6d5d3de8a82 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 12 17:19:41.463398 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:41.463398 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:41.463398 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:41.463398 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:41.463398 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:41.463398 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:41.463398 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:41.463398 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:41.463398 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:41.463398 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:41.463398 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:41.463398 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:41.463398 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:41.463398 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:41.463398 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:41.464372 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:41.464372 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:41.464372 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:41.464372 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:41.464372 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:41.464372 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:41.464372 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:41.464372 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:41.464372 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:41.464372 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:41.464372 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:41.464372 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:41.464372 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:41.464372 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:41.464372 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:41.464372 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:41.465193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:41.465193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:41.465193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:41.465193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:41.465193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:41.465193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:41.465193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:41.465193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:41.465193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:41.465193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:41.465193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 12 17:19:41.465193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:41.472328 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 195/387] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Jul 12 17:19:41 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:19:41.473701 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d312b0f7-59f6-485e-83f7-a6d5d3de8a82 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:41.564125 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5cce2d8-b088-47cc-ae7b-01e1028640eb None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:41.607135 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d7d788-3228-4713-b736-8479ed842785 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:41.608564 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d7d788-3228-4713-b736-8479ed842785 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:41.609057 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d7d788-3228-4713-b736-8479ed842785 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:41.609540 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d7d788-3228-4713-b736-8479ed842785 None admin] PATH_INFO: `/v3/groups/92246a55b7fd43469a1df54e34c8099f` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:41.611030 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d7d788-3228-4713-b736-8479ed842785 None admin] RBAC: Authorizing `identity:get_group(group_id=92246a55b7fd43469a1df54e34c8099f)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:41.612793 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d7d788-3228-4713-b736-8479ed842785 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:41.615000 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 193/388] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Jul 12 17:19:41 2022] GET /identity/v3/groups/92246a55b7fd43469a1df54e34c8099f => generated 226 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:19:41.621115 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d7d788-3228-4713-b736-8479ed842785 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:41.651605 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:41.657281 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:41.668410 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d312b0f7-59f6-485e-83f7-a6d5d3de8a82 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:41.734808 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d7d788-3228-4713-b736-8479ed842785 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:41.749425 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fefe75d-6958-4ebf-9c7c-648b60e00396 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:41.750103 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fefe75d-6958-4ebf-9c7c-648b60e00396 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:41.750389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fefe75d-6958-4ebf-9c7c-648b60e00396 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:41.750697 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fefe75d-6958-4ebf-9c7c-648b60e00396 None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:41.751469 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fefe75d-6958-4ebf-9c7c-648b60e00396 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:41.754740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fefe75d-6958-4ebf-9c7c-648b60e00396 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:41.756229 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fefe75d-6958-4ebf-9c7c-648b60e00396 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:41.775440 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 196/389] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:19:41 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:19:41.781249 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fefe75d-6958-4ebf-9c7c-648b60e00396 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:41.804334 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e5135b-3755-47f4-adde-399081fa7cd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:41.805248 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e5135b-3755-47f4-adde-399081fa7cd5 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:41.805497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e5135b-3755-47f4-adde-399081fa7cd5 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:41.805772 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e5135b-3755-47f4-adde-399081fa7cd5 None admin] PATH_INFO: `/v3/projects/4857259d2a364dff960547dc0aac75e0` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:41.808451 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81e5135b-3755-47f4-adde-399081fa7cd5 None admin] RBAC: Authorizing `identity:get_project(project_id=4857259d2a364dff960547dc0aac75e0)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:41.809905 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81e5135b-3755-47f4-adde-399081fa7cd5 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:41.812952 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 194/390] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:41 2022] GET /identity/v3/projects/4857259d2a364dff960547dc0aac75e0 => generated 298 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:19:41.819151 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e5135b-3755-47f4-adde-399081fa7cd5 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:41.987601 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fefe75d-6958-4ebf-9c7c-648b60e00396 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:42.015067 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:42.024428 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e5135b-3755-47f4-adde-399081fa7cd5 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:42.059076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7823a4-8de8-4557-905b-b427dd54079f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:42.059076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7823a4-8de8-4557-905b-b427dd54079f None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:42.059377 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7823a4-8de8-4557-905b-b427dd54079f None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:42.059571 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7823a4-8de8-4557-905b-b427dd54079f None admin] PATH_INFO: `/v3/groups/92246a55b7fd43469a1df54e34c8099f` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:42.060891 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7823a4-8de8-4557-905b-b427dd54079f None admin] RBAC: Authorizing `identity:get_group(group_id=92246a55b7fd43469a1df54e34c8099f)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:42.062475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7823a4-8de8-4557-905b-b427dd54079f None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:42.064445 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 197/391] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Jul 12 17:19:41 2022] GET /identity/v3/groups/92246a55b7fd43469a1df54e34c8099f => generated 226 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:19:42.072315 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7823a4-8de8-4557-905b-b427dd54079f None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:42.085455 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d1ee5b-be7a-4619-ab73-631c8fc66639 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:42.087894 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d1ee5b-be7a-4619-ab73-631c8fc66639 None admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:42.088407 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d1ee5b-be7a-4619-ab73-631c8fc66639 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:42.088870 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d1ee5b-be7a-4619-ab73-631c8fc66639 None admin] PATH_INFO: `/v3/projects/4857259d2a364dff960547dc0aac75e0/groups/92246a55b7fd43469a1df54e34c8099f/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:42.092717 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3d1ee5b-be7a-4619-ab73-631c8fc66639 None admin] RBAC: Authorizing `identity:create_grant(project_id=4857259d2a364dff960547dc0aac75e0, group_id=92246a55b7fd43469a1df54e34c8099f, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:42.094612 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3d1ee5b-be7a-4619-ab73-631c8fc66639 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:42.129557 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:42.184457 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 195/392] 10.222.0.16 () {58 vars in 1442 bytes} [Tue Jul 12 17:19:41 2022] PUT /identity/v3/projects/4857259d2a364dff960547dc0aac75e0/groups/92246a55b7fd43469a1df54e34c8099f/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 366 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:19:42.225698 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7823a4-8de8-4557-905b-b427dd54079f None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:42.280611 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74754ba-1f1d-494d-b64e-140a4c77ff64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:42.281847 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74754ba-1f1d-494d-b64e-140a4c77ff64 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:42.282321 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74754ba-1f1d-494d-b64e-140a4c77ff64 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:42.282828 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74754ba-1f1d-494d-b64e-140a4c77ff64 None admin] PATH_INFO: `/v3/projects/4857259d2a364dff960547dc0aac75e0` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:42.285178 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c74754ba-1f1d-494d-b64e-140a4c77ff64 None admin] RBAC: Authorizing `identity:get_project(project_id=4857259d2a364dff960547dc0aac75e0)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:42.287004 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c74754ba-1f1d-494d-b64e-140a4c77ff64 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:42.289230 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 198/393] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:42 2022] GET /identity/v3/projects/4857259d2a364dff960547dc0aac75e0 => generated 298 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:19:42.295487 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d1ee5b-be7a-4619-ab73-631c8fc66639 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:42.422105 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d1ee5b-be7a-4619-ab73-631c8fc66639 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:42.466704 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1563e75-380f-48c4-bcb5-4146ba8a821d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:42.468059 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1563e75-380f-48c4-bcb5-4146ba8a821d None admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:42.468529 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1563e75-380f-48c4-bcb5-4146ba8a821d None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:42.468987 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1563e75-380f-48c4-bcb5-4146ba8a821d None admin] PATH_INFO: `/v3/projects/4857259d2a364dff960547dc0aac75e0/groups/92246a55b7fd43469a1df54e34c8099f/roles/0c239663092d459e8c9724963e78da76` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:42.476637 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1563e75-380f-48c4-bcb5-4146ba8a821d None admin] RBAC: Authorizing `identity:create_grant(project_id=4857259d2a364dff960547dc0aac75e0, group_id=92246a55b7fd43469a1df54e34c8099f, role_id=0c239663092d459e8c9724963e78da76)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:42.478384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1563e75-380f-48c4-bcb5-4146ba8a821d None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:42.575274 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 196/394] 10.222.0.16 () {58 vars in 1442 bytes} [Tue Jul 12 17:19:42 2022] PUT /identity/v3/projects/4857259d2a364dff960547dc0aac75e0/groups/92246a55b7fd43469a1df54e34c8099f/roles/0c239663092d459e8c9724963e78da76 => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:19:43.213942 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3c6100-4ee8-4c18-a666-5fded200acc2 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:43.215031 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3c6100-4ee8-4c18-a666-5fded200acc2 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:43.215532 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3c6100-4ee8-4c18-a666-5fded200acc2 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:43.217504 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 199/395] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:43 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:43.231399 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baea86e2-0b31-4458-b92a-329ab869488b None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:43.232393 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baea86e2-0b31-4458-b92a-329ab869488b None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:43.232963 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baea86e2-0b31-4458-b92a-329ab869488b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:43.259697 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-baea86e2-0b31-4458-b92a-329ab869488b None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:43.261322 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-baea86e2-0b31-4458-b92a-329ab869488b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:43.271762 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:43.347876 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 197/396] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:43.386208 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3c6100-4ee8-4c18-a666-5fded200acc2 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:43.465439 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad5e850-dbc4-455c-bde3-5629b4a378ce None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:43.465751 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad5e850-dbc4-455c-bde3-5629b4a378ce None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:43.465951 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ad5e850-dbc4-455c-bde3-5629b4a378ce None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:43.467349 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 198/397] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:43.488094 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f70ed3a-9096-497d-bfd7-ecbf8cca475d None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:43.488407 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f70ed3a-9096-497d-bfd7-ecbf8cca475d None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:43.488673 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f70ed3a-9096-497d-bfd7-ecbf8cca475d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:43.522845 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:43.527931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3c6100-4ee8-4c18-a666-5fded200acc2 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:43.542144 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-2f70ed3a-9096-497d-bfd7-ecbf8cca475d None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:43.543477 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-2f70ed3a-9096-497d-bfd7-ecbf8cca475d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:43.601369 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb91e434-faab-4b41-afe5-9109fe5a17bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:43.604064 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb91e434-faab-4b41-afe5-9109fe5a17bf None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:43.604353 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb91e434-faab-4b41-afe5-9109fe5a17bf None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:43.605682 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb91e434-faab-4b41-afe5-9109fe5a17bf None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:43.628385 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb91e434-faab-4b41-afe5-9109fe5a17bf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:43.637965 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-fb91e434-faab-4b41-afe5-9109fe5a17bf None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 12 17:19:43.637965 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:43.637965 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:43.637965 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:43.637965 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:43.637965 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:43.637965 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:43.637965 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:43.637965 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:43.637965 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:43.637965 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:43.637965 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:43.637965 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:43.637965 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:43.637965 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:43.637965 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:43.638930 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:43.638930 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:43.638930 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:43.638930 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:43.638930 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:43.638930 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:43.638930 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:43.638930 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:43.638930 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:43.638930 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:43.638930 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:43.638930 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:43.638930 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:43.638930 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:43.638930 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:43.638930 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:43.639718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:43.639718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:43.639718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:43.639718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:43.639718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:43.639718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:43.639718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:43.639718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:43.639718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:43.639718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:43.639718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 12 17:19:43.639718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:43.650749 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 200/398] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Jul 12 17:19:43 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:43.654811 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb91e434-faab-4b41-afe5-9109fe5a17bf None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:43.674463 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 199/399] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 363 bytes (3 switches on core 0) Jul 12 17:19:43.739016 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f70ed3a-9096-497d-bfd7-ecbf8cca475d None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:43.828713 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb91e434-faab-4b41-afe5-9109fe5a17bf None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:43.842152 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:43.875504 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f70ed3a-9096-497d-bfd7-ecbf8cca475d None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:43.880022 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965c7881-6973-48cc-b727-ebb1b153ea1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:43.880022 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965c7881-6973-48cc-b727-ebb1b153ea1b None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:43.880348 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965c7881-6973-48cc-b727-ebb1b153ea1b None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:43.880594 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965c7881-6973-48cc-b727-ebb1b153ea1b None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:43.881353 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-965c7881-6973-48cc-b727-ebb1b153ea1b None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:43.881671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-965c7881-6973-48cc-b727-ebb1b153ea1b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:43.883094 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-965c7881-6973-48cc-b727-ebb1b153ea1b None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:43.892274 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 201/400] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:19:43 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:43.895237 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01dbdb89-8b1c-4b38-92b9-6fd6e7cec8ff None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:43.895747 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01dbdb89-8b1c-4b38-92b9-6fd6e7cec8ff None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:43.896416 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01dbdb89-8b1c-4b38-92b9-6fd6e7cec8ff None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:43.898230 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 202/401] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:43 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:43.902389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01dbdb89-8b1c-4b38-92b9-6fd6e7cec8ff None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:43.920051 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:43.927508 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b730453b-a7d0-423a-a59c-0339ce7a46e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:43.928591 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b730453b-a7d0-423a-a59c-0339ce7a46e5 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:43.932686 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b730453b-a7d0-423a-a59c-0339ce7a46e5 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:43.932686 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b730453b-a7d0-423a-a59c-0339ce7a46e5 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:43.945032 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b730453b-a7d0-423a-a59c-0339ce7a46e5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:43.954839 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-b730453b-a7d0-423a-a59c-0339ce7a46e5 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 12 17:19:43.954839 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:43.954839 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:43.954839 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:43.954839 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:43.954839 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:43.954839 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:43.954839 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:43.954839 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:43.954839 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:43.954839 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:43.954839 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:43.954839 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:43.954839 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:43.954839 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:43.954839 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:43.956354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:43.956354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:43.956354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:43.956354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:43.956354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:43.956354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:43.956354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:43.956354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:43.956354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:43.956354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:43.956354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:43.956354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:43.956354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:43.956354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:43.956354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:43.956354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:43.957163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:43.957163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:43.957163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:43.957163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:43.957163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:43.957163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:43.957163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:43.957163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:43.957163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:43.957163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:43.957163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 12 17:19:43.957163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:19:43.964858 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 200/402] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:43 2022] GET /identity/v3/roles/member => generated 84 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:19:43.964858 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d1148e7-6718-486e-9cab-aa5229dea503 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:43.964858 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d1148e7-6718-486e-9cab-aa5229dea503 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:43.964858 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d1148e7-6718-486e-9cab-aa5229dea503 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:43.987304 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-0d1148e7-6718-486e-9cab-aa5229dea503 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:43.989493 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-0d1148e7-6718-486e-9cab-aa5229dea503 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:44.026479 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01dbdb89-8b1c-4b38-92b9-6fd6e7cec8ff None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:44.065434 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5cdfb2-29f2-42f7-b10f-f66796f1d07c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:44.066378 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5cdfb2-29f2-42f7-b10f-f66796f1d07c None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:44.066678 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5cdfb2-29f2-42f7-b10f-f66796f1d07c None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:44.066963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5cdfb2-29f2-42f7-b10f-f66796f1d07c None admin] PATH_INFO: `/v3/groups/92246a55b7fd43469a1df54e34c8099f` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:44.068212 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e5cdfb2-29f2-42f7-b10f-f66796f1d07c None admin] RBAC: Authorizing `identity:get_group(group_id=92246a55b7fd43469a1df54e34c8099f)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:44.069712 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e5cdfb2-29f2-42f7-b10f-f66796f1d07c None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:44.071794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 203/403] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Jul 12 17:19:43 2022] GET /identity/v3/groups/92246a55b7fd43469a1df54e34c8099f => generated 226 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:44.072743 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5cdfb2-29f2-42f7-b10f-f66796f1d07c None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:44.086958 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 201/404] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:44.087490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d1148e7-6718-486e-9cab-aa5229dea503 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:44.166668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:44.186622 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:44.204238 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5cdfb2-29f2-42f7-b10f-f66796f1d07c None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:44.218843 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d1148e7-6718-486e-9cab-aa5229dea503 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:44.242988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cbcd43e-64f1-4f19-a4cc-6277b8982d83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:44.243596 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbcd43e-64f1-4f19-a4cc-6277b8982d83 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:44.244033 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbcd43e-64f1-4f19-a4cc-6277b8982d83 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:44.244395 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbcd43e-64f1-4f19-a4cc-6277b8982d83 None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:44.245214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cbcd43e-64f1-4f19-a4cc-6277b8982d83 None admin] RBAC: Adding query filter params (name=member) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:44.245614 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cbcd43e-64f1-4f19-a4cc-6277b8982d83 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:44.247184 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cbcd43e-64f1-4f19-a4cc-6277b8982d83 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:44.261469 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 204/405] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Jul 12 17:19:44 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:44.261469 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cbcd43e-64f1-4f19-a4cc-6277b8982d83 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:44.273667 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2601d0-079f-42d1-82ee-3c8e49eb0824 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:44.275139 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2601d0-079f-42d1-82ee-3c8e49eb0824 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:44.275600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2601d0-079f-42d1-82ee-3c8e49eb0824 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:44.276413 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2601d0-079f-42d1-82ee-3c8e49eb0824 None admin] PATH_INFO: `/v3/projects/31b9ba63c2994be69d5ff224d61a9c25` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:44.278204 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2601d0-079f-42d1-82ee-3c8e49eb0824 None admin] RBAC: Authorizing `identity:get_project(project_id=31b9ba63c2994be69d5ff224d61a9c25)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:44.280129 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2601d0-079f-42d1-82ee-3c8e49eb0824 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:44.282718 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 202/406] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:44 2022] GET /identity/v3/projects/31b9ba63c2994be69d5ff224d61a9c25 => generated 302 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:44.284684 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2601d0-079f-42d1-82ee-3c8e49eb0824 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:44.379284 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cbcd43e-64f1-4f19-a4cc-6277b8982d83 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:44.416975 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf8ff46-a328-4d7e-af30-0185cbeb9885 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:44.417908 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf8ff46-a328-4d7e-af30-0185cbeb9885 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:44.418264 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf8ff46-a328-4d7e-af30-0185cbeb9885 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:44.418661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf8ff46-a328-4d7e-af30-0185cbeb9885 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:44.422637 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2601d0-079f-42d1-82ee-3c8e49eb0824 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:44.432831 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abf8ff46-a328-4d7e-af30-0185cbeb9885 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:44.444210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-abf8ff46-a328-4d7e-af30-0185cbeb9885 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:19:44.444210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:44.444210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:44.444210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:44.444210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:44.444210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:44.444210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:44.444210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:44.444210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:44.444210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:44.444210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:44.444210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:44.444210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:44.444210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:44.444210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:44.444210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:44.445142 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:44.445142 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:44.445142 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:44.445142 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:44.445142 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:44.445142 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:44.445142 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:44.445142 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:44.445142 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:44.445142 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:44.445142 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:44.445142 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:44.445142 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:44.445142 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:44.445142 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:44.445142 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:44.445972 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:44.445972 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:44.445972 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:44.445972 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:44.445972 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:44.445972 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:44.445972 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:44.445972 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:44.445972 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:44.445972 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:44.445972 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:19:44.445972 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:44.452824 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 205/407] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Jul 12 17:19:44 2022] GET /identity/v3/roles/admin => generated 83 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:44.453754 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf8ff46-a328-4d7e-af30-0185cbeb9885 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:44.458028 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:44.464704 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:44.465833 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b129c4-12c3-4205-8423-e97f1c0fee83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:44.467206 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b129c4-12c3-4205-8423-e97f1c0fee83 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:44.474123 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b129c4-12c3-4205-8423-e97f1c0fee83 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:44.474123 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b129c4-12c3-4205-8423-e97f1c0fee83 None admin] PATH_INFO: `/v3/groups/92246a55b7fd43469a1df54e34c8099f` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:44.474123 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b129c4-12c3-4205-8423-e97f1c0fee83 None admin] RBAC: Authorizing `identity:get_group(group_id=92246a55b7fd43469a1df54e34c8099f)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:44.474123 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20b129c4-12c3-4205-8423-e97f1c0fee83 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:44.474123 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 203/408] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Jul 12 17:19:44 2022] GET /identity/v3/groups/92246a55b7fd43469a1df54e34c8099f => generated 226 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:44.475174 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b129c4-12c3-4205-8423-e97f1c0fee83 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:44.595605 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf8ff46-a328-4d7e-af30-0185cbeb9885 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:44.602585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b129c4-12c3-4205-8423-e97f1c0fee83 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:44.641279 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e3aac25-0855-4b32-a281-35d929633c30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:44.642183 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3aac25-0855-4b32-a281-35d929633c30 None admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:44.643076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3aac25-0855-4b32-a281-35d929633c30 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:44.643315 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-790e5d0e-ec28-4a52-80cd-4d712b52b44d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:44.643429 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3aac25-0855-4b32-a281-35d929633c30 None admin] PATH_INFO: `/v3/projects/31b9ba63c2994be69d5ff224d61a9c25/groups/92246a55b7fd43469a1df54e34c8099f/roles/0c239663092d459e8c9724963e78da76` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:44.644274 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790e5d0e-ec28-4a52-80cd-4d712b52b44d None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:44.644753 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790e5d0e-ec28-4a52-80cd-4d712b52b44d None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:44.645205 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790e5d0e-ec28-4a52-80cd-4d712b52b44d None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:44.645686 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e3aac25-0855-4b32-a281-35d929633c30 None admin] RBAC: Authorizing `identity:create_grant(project_id=31b9ba63c2994be69d5ff224d61a9c25, group_id=92246a55b7fd43469a1df54e34c8099f, role_id=0c239663092d459e8c9724963e78da76)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:44.646330 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-790e5d0e-ec28-4a52-80cd-4d712b52b44d None admin] RBAC: Adding query filter params (name=admin) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:44.646846 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-790e5d0e-ec28-4a52-80cd-4d712b52b44d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:44.647191 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e3aac25-0855-4b32-a281-35d929633c30 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:44.648564 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-790e5d0e-ec28-4a52-80cd-4d712b52b44d None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:44.664711 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 204/409] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:44 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:44.664711 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-790e5d0e-ec28-4a52-80cd-4d712b52b44d None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:44.728708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:44.746649 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 206/410] 10.222.0.16 () {58 vars in 1442 bytes} [Tue Jul 12 17:19:44 2022] PUT /identity/v3/projects/31b9ba63c2994be69d5ff224d61a9c25/groups/92246a55b7fd43469a1df54e34c8099f/roles/0c239663092d459e8c9724963e78da76 => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:19:44.748383 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e3aac25-0855-4b32-a281-35d929633c30 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:44.775645 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-790e5d0e-ec28-4a52-80cd-4d712b52b44d None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:44.818517 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82887e3-212f-45cd-982b-06124889def2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:44.819417 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82887e3-212f-45cd-982b-06124889def2 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:44.819751 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82887e3-212f-45cd-982b-06124889def2 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:44.820039 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82887e3-212f-45cd-982b-06124889def2 None admin] PATH_INFO: `/v3/projects/31b9ba63c2994be69d5ff224d61a9c25` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:44.821470 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82887e3-212f-45cd-982b-06124889def2 None admin] RBAC: Authorizing `identity:get_project(project_id=31b9ba63c2994be69d5ff224d61a9c25)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:44.823388 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82887e3-212f-45cd-982b-06124889def2 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:44.825136 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 205/411] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:44 2022] GET /identity/v3/projects/31b9ba63c2994be69d5ff224d61a9c25 => generated 302 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:44.832423 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e845dcbc-b125-429f-ab83-197205f89fb3 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:44.832423 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e845dcbc-b125-429f-ab83-197205f89fb3 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:44.832815 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e845dcbc-b125-429f-ab83-197205f89fb3 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:44.834923 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 206/412] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:44 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:44.835689 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e845dcbc-b125-429f-ab83-197205f89fb3 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:44.852133 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:44.898424 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e3aac25-0855-4b32-a281-35d929633c30 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:44.940058 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63653465-e1f2-4f2f-b6f6-2e0c920d467e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:44.940942 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63653465-e1f2-4f2f-b6f6-2e0c920d467e None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:44.941233 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63653465-e1f2-4f2f-b6f6-2e0c920d467e None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:44.941534 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63653465-e1f2-4f2f-b6f6-2e0c920d467e None admin] PATH_INFO: `/v3/groups/059e96df72874559ab13e17f5c9d5cfc` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:44.942803 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63653465-e1f2-4f2f-b6f6-2e0c920d467e None admin] RBAC: Authorizing `identity:get_group(group_id=059e96df72874559ab13e17f5c9d5cfc)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:44.944724 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63653465-e1f2-4f2f-b6f6-2e0c920d467e None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:44.946057 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 207/413] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Jul 12 17:19:44 2022] GET /identity/v3/groups/059e96df72874559ab13e17f5c9d5cfc => generated 229 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:44.948292 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80052b9-a963-4185-9948-8021b3951b23 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:44.948609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80052b9-a963-4185-9948-8021b3951b23 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:44.948872 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80052b9-a963-4185-9948-8021b3951b23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:44.966182 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-e80052b9-a963-4185-9948-8021b3951b23 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:44.967418 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-e80052b9-a963-4185-9948-8021b3951b23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:44.980768 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e845dcbc-b125-429f-ab83-197205f89fb3 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:45.012773 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:45.021610 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308c45b0-964a-4293-ad63-2992d7bcc9de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:45.022522 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308c45b0-964a-4293-ad63-2992d7bcc9de None admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:45.022811 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308c45b0-964a-4293-ad63-2992d7bcc9de None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:45.023310 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308c45b0-964a-4293-ad63-2992d7bcc9de None admin] PATH_INFO: `/v3/projects/31b9ba63c2994be69d5ff224d61a9c25/groups/92246a55b7fd43469a1df54e34c8099f/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:45.025220 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-308c45b0-964a-4293-ad63-2992d7bcc9de None admin] RBAC: Authorizing `identity:create_grant(project_id=31b9ba63c2994be69d5ff224d61a9c25, group_id=92246a55b7fd43469a1df54e34c8099f, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:45.026682 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-308c45b0-964a-4293-ad63-2992d7bcc9de None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:45.043693 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 208/414] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:45.044547 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80052b9-a963-4185-9948-8021b3951b23 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:45.114313 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:45.130460 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 207/415] 10.222.0.16 () {58 vars in 1442 bytes} [Tue Jul 12 17:19:44 2022] PUT /identity/v3/projects/31b9ba63c2994be69d5ff224d61a9c25/groups/92246a55b7fd43469a1df54e34c8099f/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 286 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:19:45.130460 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb50459-83b1-42fc-b563-206f03adf1e5 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:45.130460 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb50459-83b1-42fc-b563-206f03adf1e5 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:45.130460 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb50459-83b1-42fc-b563-206f03adf1e5 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:45.130460 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 208/416] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:45.130460 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb50459-83b1-42fc-b563-206f03adf1e5 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:45.181059 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80052b9-a963-4185-9948-8021b3951b23 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:45.233717 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9238f43d-e524-4695-83fa-3b42f1f1bae8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:45.234926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9238f43d-e524-4695-83fa-3b42f1f1bae8 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:45.235221 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9238f43d-e524-4695-83fa-3b42f1f1bae8 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:45.235509 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9238f43d-e524-4695-83fa-3b42f1f1bae8 None admin] PATH_INFO: `/v3/projects/1d1f69915b004b66af4c48b7c59a15d8` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:45.237592 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9238f43d-e524-4695-83fa-3b42f1f1bae8 None admin] RBAC: Authorizing `identity:get_project(project_id=1d1f69915b004b66af4c48b7c59a15d8)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:45.239241 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9238f43d-e524-4695-83fa-3b42f1f1bae8 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:45.242123 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 209/417] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:45 2022] GET /identity/v3/projects/1d1f69915b004b66af4c48b7c59a15d8 => generated 344 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:45.243867 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90fcb63-fccc-4fbd-8cbb-f05f2970ed2c None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:45.244400 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90fcb63-fccc-4fbd-8cbb-f05f2970ed2c None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:45.244710 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90fcb63-fccc-4fbd-8cbb-f05f2970ed2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:45.258570 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb50459-83b1-42fc-b563-206f03adf1e5 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:45.268766 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-a90fcb63-fccc-4fbd-8cbb-f05f2970ed2c None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:45.270202 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-a90fcb63-fccc-4fbd-8cbb-f05f2970ed2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:45.309839 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9160d287-21b2-4e02-b219-31a8c4219a87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:45.310677 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9160d287-21b2-4e02-b219-31a8c4219a87 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:45.310970 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9160d287-21b2-4e02-b219-31a8c4219a87 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:45.311275 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9160d287-21b2-4e02-b219-31a8c4219a87 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:45.327657 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9160d287-21b2-4e02-b219-31a8c4219a87 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:45.336319 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-9160d287-21b2-4e02-b219-31a8c4219a87 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 12 17:19:45.336319 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:45.336319 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:45.336319 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:45.336319 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:45.336319 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:45.336319 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:45.336319 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:45.336319 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:45.336319 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:45.336319 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:45.336319 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:45.336319 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:45.336319 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:45.336319 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:45.336319 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:45.337241 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:45.337241 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:45.337241 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:45.337241 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:45.337241 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:45.337241 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:45.337241 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:45.337241 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:45.337241 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:45.337241 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:45.337241 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:45.337241 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:45.337241 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:45.337241 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:45.337241 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:45.337241 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:45.338043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:45.338043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:45.338043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:45.338043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:45.338043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:45.338043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:45.338043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:45.338043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:45.338043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:45.338043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:45.338043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 12 17:19:45.338043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:19:45.338043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 209/418] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:45 2022] GET /identity/v3/roles/member => generated 84 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:45.338745 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9160d287-21b2-4e02-b219-31a8c4219a87 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:45.361347 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 210/419] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:45.362133 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a90fcb63-fccc-4fbd-8cbb-f05f2970ed2c None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:45.415828 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:45.430760 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:45.473030 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9160d287-21b2-4e02-b219-31a8c4219a87 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:45.475463 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a90fcb63-fccc-4fbd-8cbb-f05f2970ed2c None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:45.515815 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7abeb6b8-a814-49fa-80fc-ba320d46b814 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:45.516265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e95a9a-279c-4489-9121-31178dd5c46f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:45.516491 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abeb6b8-a814-49fa-80fc-ba320d46b814 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:45.516777 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abeb6b8-a814-49fa-80fc-ba320d46b814 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:45.517122 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abeb6b8-a814-49fa-80fc-ba320d46b814 None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:45.517285 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e95a9a-279c-4489-9121-31178dd5c46f None admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:45.518012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e95a9a-279c-4489-9121-31178dd5c46f None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:45.522466 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e95a9a-279c-4489-9121-31178dd5c46f None admin] PATH_INFO: `/v3/projects/1d1f69915b004b66af4c48b7c59a15d8/groups/059e96df72874559ab13e17f5c9d5cfc/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:45.522703 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7abeb6b8-a814-49fa-80fc-ba320d46b814 None admin] RBAC: Adding query filter params (name=member) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:45.522703 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7abeb6b8-a814-49fa-80fc-ba320d46b814 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:45.522995 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e95a9a-279c-4489-9121-31178dd5c46f None admin] RBAC: Authorizing `identity:create_grant(project_id=1d1f69915b004b66af4c48b7c59a15d8, group_id=059e96df72874559ab13e17f5c9d5cfc, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:45.523141 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7abeb6b8-a814-49fa-80fc-ba320d46b814 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:45.523328 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e95a9a-279c-4489-9121-31178dd5c46f None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:45.530355 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 211/420] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Jul 12 17:19:45 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:45.531337 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7abeb6b8-a814-49fa-80fc-ba320d46b814 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:45.596779 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 210/421] 10.222.0.16 () {58 vars in 1442 bytes} [Tue Jul 12 17:19:45 2022] PUT /identity/v3/projects/1d1f69915b004b66af4c48b7c59a15d8/groups/059e96df72874559ab13e17f5c9d5cfc/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:19:45.598404 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e95a9a-279c-4489-9121-31178dd5c46f None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:45.696960 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7abeb6b8-a814-49fa-80fc-ba320d46b814 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:45.730766 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e95a9a-279c-4489-9121-31178dd5c46f None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:45.742830 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:45.751610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:45.767852 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f631a2e-9634-4850-a405-87edc6def4ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:45.768684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f631a2e-9634-4850-a405-87edc6def4ab None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:45.769027 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f631a2e-9634-4850-a405-87edc6def4ab None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:45.769301 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f631a2e-9634-4850-a405-87edc6def4ab None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:45.772114 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd7143a-afec-460e-9e18-abf4825edd6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:45.772778 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd7143a-afec-460e-9e18-abf4825edd6d None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:45.773069 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd7143a-afec-460e-9e18-abf4825edd6d None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:45.773350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd7143a-afec-460e-9e18-abf4825edd6d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:45.782625 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f631a2e-9634-4850-a405-87edc6def4ab None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:45.792282 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-5f631a2e-9634-4850-a405-87edc6def4ab None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 12 17:19:45.792282 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:45.792282 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:45.792282 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:45.792282 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:45.792282 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:45.792282 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:45.792282 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:45.792282 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:45.792282 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:45.792282 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:45.792282 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:45.792282 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:45.792282 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:45.792282 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:45.792282 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:45.800216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:45.800216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:45.800216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:45.800216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:45.800216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:45.800216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:45.800216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:45.800216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:45.800216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:45.800216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:45.800216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:45.800216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:45.800216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:45.800216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:45.800216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:45.800216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:45.801012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:45.801012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:45.801012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:45.801012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:45.801012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:45.801012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:45.801012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:45.801012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:45.801012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:45.801012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:45.801012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 12 17:19:45.801012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:45.801012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 212/422] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Jul 12 17:19:45 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 263 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:45.801688 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f631a2e-9634-4850-a405-87edc6def4ab None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:45.875725 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd7143a-afec-460e-9e18-abf4825edd6d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:45.877480 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd7143a-afec-460e-9e18-abf4825edd6d None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:45.983344 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f631a2e-9634-4850-a405-87edc6def4ab None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:45.984294 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 211/423] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:19:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 12 17:19:45.992327 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd7143a-afec-460e-9e18-abf4825edd6d None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:46.004202 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:46.038599 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badc25dd-4e58-48f7-8e33-f8fa0c12a110 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:46.039276 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badc25dd-4e58-48f7-8e33-f8fa0c12a110 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:46.039559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badc25dd-4e58-48f7-8e33-f8fa0c12a110 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:46.039839 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badc25dd-4e58-48f7-8e33-f8fa0c12a110 None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:46.040590 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-badc25dd-4e58-48f7-8e33-f8fa0c12a110 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:46.040898 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-badc25dd-4e58-48f7-8e33-f8fa0c12a110 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:46.044912 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-badc25dd-4e58-48f7-8e33-f8fa0c12a110 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:46.070572 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 213/424] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:19:45 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:46.080606 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badc25dd-4e58-48f7-8e33-f8fa0c12a110 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:46.136575 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:46.193774 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badc25dd-4e58-48f7-8e33-f8fa0c12a110 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:46.242860 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd7143a-afec-460e-9e18-abf4825edd6d None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:46.256070 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf3480f-1601-4233-9209-d14dd9c64a61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:46.256747 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf3480f-1601-4233-9209-d14dd9c64a61 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:46.257057 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf3480f-1601-4233-9209-d14dd9c64a61 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:46.257338 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf3480f-1601-4233-9209-d14dd9c64a61 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:46.311286 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc1dd77-8a5a-4372-88b9-5ec77ab0faa4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:46.312163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc1dd77-8a5a-4372-88b9-5ec77ab0faa4 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:46.312447 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc1dd77-8a5a-4372-88b9-5ec77ab0faa4 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:46.312722 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc1dd77-8a5a-4372-88b9-5ec77ab0faa4 None admin] PATH_INFO: `/v3/projects/4857259d2a364dff960547dc0aac75e0` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:46.315599 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc1dd77-8a5a-4372-88b9-5ec77ab0faa4 None admin] RBAC: Authorizing `identity:get_project(project_id=4857259d2a364dff960547dc0aac75e0)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:46.318373 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc1dd77-8a5a-4372-88b9-5ec77ab0faa4 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:46.320322 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 212/425] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:45 2022] GET /identity/v3/projects/4857259d2a364dff960547dc0aac75e0 => generated 298 bytes in 328 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:46.325774 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc1dd77-8a5a-4372-88b9-5ec77ab0faa4 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:46.397388 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf3480f-1601-4233-9209-d14dd9c64a61 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:46.407006 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf3480f-1601-4233-9209-d14dd9c64a61 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:46.410293 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:46.456855 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc1dd77-8a5a-4372-88b9-5ec77ab0faa4 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:46.458780 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:46.529983 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bbfd2ee-1779-4f0c-8fa4-ccde4748b709 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:46.530793 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bbfd2ee-1779-4f0c-8fa4-ccde4748b709 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:46.531082 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bbfd2ee-1779-4f0c-8fa4-ccde4748b709 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:46.531478 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bbfd2ee-1779-4f0c-8fa4-ccde4748b709 None admin] PATH_INFO: `/v3/groups/92246a55b7fd43469a1df54e34c8099f` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:46.533867 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bbfd2ee-1779-4f0c-8fa4-ccde4748b709 None admin] RBAC: Authorizing `identity:get_group(group_id=92246a55b7fd43469a1df54e34c8099f)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:46.535352 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bbfd2ee-1779-4f0c-8fa4-ccde4748b709 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:46.539637 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 213/426] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Jul 12 17:19:46 2022] GET /identity/v3/groups/92246a55b7fd43469a1df54e34c8099f => generated 226 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:46.550628 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bbfd2ee-1779-4f0c-8fa4-ccde4748b709 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:46.555372 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 214/427] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:19:46 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 476 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 12 17:19:46.568406 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf3480f-1601-4233-9209-d14dd9c64a61 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:46.664545 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bbfd2ee-1779-4f0c-8fa4-ccde4748b709 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:46.699912 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee03d7b7-c44f-494f-9ea6-7430c7875092 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:46.700610 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee03d7b7-c44f-494f-9ea6-7430c7875092 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:46.700895 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee03d7b7-c44f-494f-9ea6-7430c7875092 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:46.701170 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee03d7b7-c44f-494f-9ea6-7430c7875092 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:46.702004 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee03d7b7-c44f-494f-9ea6-7430c7875092 None admin] RBAC: Adding query filter params (group.id=92246a55b7fd43469a1df54e34c8099f, role.id=4c5d52f873f94dc1b10918068376add5, scope.project.id=4857259d2a364dff960547dc0aac75e0) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:46.702313 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee03d7b7-c44f-494f-9ea6-7430c7875092 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:46.703801 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee03d7b7-c44f-494f-9ea6-7430c7875092 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:46.710332 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 214/428] 10.222.0.16 () {56 vars in 1376 bytes} [Tue Jul 12 17:19:46 2022] GET /identity/v3/role_assignments?group.id=92246a55b7fd43469a1df54e34c8099f&scope.project.id=4857259d2a364dff960547dc0aac75e0&role.id=4c5d52f873f94dc1b10918068376add5 => generated 611 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:46.813544 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf3480f-1601-4233-9209-d14dd9c64a61 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:46.869373 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ac67cc-a377-4b05-8a90-810bde750cb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:46.870251 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ac67cc-a377-4b05-8a90-810bde750cb4 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:46.870563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ac67cc-a377-4b05-8a90-810bde750cb4 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:46.870845 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ac67cc-a377-4b05-8a90-810bde750cb4 None admin] PATH_INFO: `/v3/projects/4857259d2a364dff960547dc0aac75e0` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:46.872061 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8ac67cc-a377-4b05-8a90-810bde750cb4 None admin] RBAC: Authorizing `identity:get_project(project_id=4857259d2a364dff960547dc0aac75e0)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:46.873514 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8ac67cc-a377-4b05-8a90-810bde750cb4 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:46.875740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 215/429] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:46 2022] GET /identity/v3/projects/4857259d2a364dff960547dc0aac75e0 => generated 298 bytes in 308 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:19:46.881365 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee03d7b7-c44f-494f-9ea6-7430c7875092 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:46.913166 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:47.036635 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee03d7b7-c44f-494f-9ea6-7430c7875092 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:47.077455 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5add550f-0351-460e-8c67-8353c94621f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:47.082467 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5add550f-0351-460e-8c67-8353c94621f9 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:47.082467 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5add550f-0351-460e-8c67-8353c94621f9 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:47.082467 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5add550f-0351-460e-8c67-8353c94621f9 None admin] PATH_INFO: `/v3/groups/92246a55b7fd43469a1df54e34c8099f` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:47.083500 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5add550f-0351-460e-8c67-8353c94621f9 None admin] RBAC: Authorizing `identity:get_group(group_id=92246a55b7fd43469a1df54e34c8099f)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:47.084991 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5add550f-0351-460e-8c67-8353c94621f9 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:47.086705 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 215/430] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Jul 12 17:19:46 2022] GET /identity/v3/groups/92246a55b7fd43469a1df54e34c8099f => generated 226 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:47.093099 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ac67cc-a377-4b05-8a90-810bde750cb4 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:47.104024 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824644ea-8087-4272-81ca-e77211ad4883 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:47.104305 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824644ea-8087-4272-81ca-e77211ad4883 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:47.104594 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824644ea-8087-4272-81ca-e77211ad4883 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:47.106367 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 216/431] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:47.113483 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d77c259-f324-482f-8a1d-89d34412f00f None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:47.113775 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d77c259-f324-482f-8a1d-89d34412f00f None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:47.114118 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d77c259-f324-482f-8a1d-89d34412f00f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:47.114759 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:47.138826 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-7d77c259-f324-482f-8a1d-89d34412f00f None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:47.140042 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-7d77c259-f324-482f-8a1d-89d34412f00f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:47.217697 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ac67cc-a377-4b05-8a90-810bde750cb4 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:47.234457 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 217/432] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 12 17:19:47.255907 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d77c259-f324-482f-8a1d-89d34412f00f None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:47.266619 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:47.273180 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5936e281-f3bf-4b3a-afb1-d38613984146 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:47.273670 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5936e281-f3bf-4b3a-afb1-d38613984146 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:47.273670 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5936e281-f3bf-4b3a-afb1-d38613984146 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:47.273920 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5936e281-f3bf-4b3a-afb1-d38613984146 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:47.275280 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5936e281-f3bf-4b3a-afb1-d38613984146 None admin] RBAC: Adding query filter params (group.id=92246a55b7fd43469a1df54e34c8099f, role.id=0c239663092d459e8c9724963e78da76, scope.project.id=4857259d2a364dff960547dc0aac75e0) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:47.275670 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5936e281-f3bf-4b3a-afb1-d38613984146 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:47.277370 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5936e281-f3bf-4b3a-afb1-d38613984146 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:47.286589 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 216/433] 10.222.0.16 () {56 vars in 1376 bytes} [Tue Jul 12 17:19:47 2022] GET /identity/v3/role_assignments?group.id=92246a55b7fd43469a1df54e34c8099f&scope.project.id=4857259d2a364dff960547dc0aac75e0&role.id=0c239663092d459e8c9724963e78da76 => generated 611 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:47.402733 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d77c259-f324-482f-8a1d-89d34412f00f None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:47.450657 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48055605-0d50-4e5c-9dc6-913950767de6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:47.451609 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48055605-0d50-4e5c-9dc6-913950767de6 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:47.451899 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48055605-0d50-4e5c-9dc6-913950767de6 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:47.452177 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48055605-0d50-4e5c-9dc6-913950767de6 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:47.465798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48055605-0d50-4e5c-9dc6-913950767de6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:47.474445 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-48055605-0d50-4e5c-9dc6-913950767de6 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 12 17:19:47.474445 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:47.474445 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:47.474445 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:47.474445 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:47.474445 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:47.474445 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:47.474445 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:47.474445 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:47.474445 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:47.474445 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:47.474445 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:47.474445 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:47.474445 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:47.474445 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:47.474445 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:47.475416 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:47.475416 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:47.475416 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:47.475416 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:47.475416 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:47.475416 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:47.475416 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:47.475416 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:47.475416 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:47.475416 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:47.475416 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:47.475416 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:47.475416 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:47.475416 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:47.475416 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:47.475416 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:47.476454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:47.476454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:47.476454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:47.476454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:47.476454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:47.476454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:47.476454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:47.476454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:47.476454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:47.476454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:47.476454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 12 17:19:47.476454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:19:47.476454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 218/434] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Jul 12 17:19:47 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:19:47.483676 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5936e281-f3bf-4b3a-afb1-d38613984146 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:47.606358 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:47.634765 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5936e281-f3bf-4b3a-afb1-d38613984146 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:47.636830 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3471e158-02f9-49e0-b9ca-19a2680d0caa None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:47.637126 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3471e158-02f9-49e0-b9ca-19a2680d0caa None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:47.637414 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3471e158-02f9-49e0-b9ca-19a2680d0caa None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:47.639080 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 219/435] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:47.644718 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56deea24-fcf3-4e97-a9fd-5be075301b4f None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:47.645046 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56deea24-fcf3-4e97-a9fd-5be075301b4f None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:47.645331 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56deea24-fcf3-4e97-a9fd-5be075301b4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:47.665053 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-56deea24-fcf3-4e97-a9fd-5be075301b4f None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:47.666167 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-56deea24-fcf3-4e97-a9fd-5be075301b4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:47.676868 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a1a1f2-bff6-4712-b1a6-685f30bee89e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:47.677518 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a1a1f2-bff6-4712-b1a6-685f30bee89e None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:47.677804 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a1a1f2-bff6-4712-b1a6-685f30bee89e None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:47.678087 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a1a1f2-bff6-4712-b1a6-685f30bee89e None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:47.678891 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a1a1f2-bff6-4712-b1a6-685f30bee89e None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:47.679206 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a1a1f2-bff6-4712-b1a6-685f30bee89e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:47.680572 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a1a1f2-bff6-4712-b1a6-685f30bee89e None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:47.694543 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 217/436] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:19:47 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:47.697142 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a1a1f2-bff6-4712-b1a6-685f30bee89e None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:47.746277 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:47.758539 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 220/437] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 12 17:19:47.781645 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56deea24-fcf3-4e97-a9fd-5be075301b4f None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:47.824955 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a1a1f2-bff6-4712-b1a6-685f30bee89e None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:47.877335 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de7195ac-3dbc-4e06-83ef-6e935e0e740e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:47.877997 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7195ac-3dbc-4e06-83ef-6e935e0e740e None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:47.878288 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7195ac-3dbc-4e06-83ef-6e935e0e740e None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:47.878623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7195ac-3dbc-4e06-83ef-6e935e0e740e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:47.892965 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:47.918169 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56deea24-fcf3-4e97-a9fd-5be075301b4f None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:47.958767 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de7195ac-3dbc-4e06-83ef-6e935e0e740e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:47.958966 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7148e39c-12cc-4994-a191-5b433e762b0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:47.959639 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7148e39c-12cc-4994-a191-5b433e762b0a None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:47.959928 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7148e39c-12cc-4994-a191-5b433e762b0a None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:47.960295 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7148e39c-12cc-4994-a191-5b433e762b0a None admin] PATH_INFO: `/v3/roles/member` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:47.960406 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de7195ac-3dbc-4e06-83ef-6e935e0e740e None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:47.972593 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7148e39c-12cc-4994-a191-5b433e762b0a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:47.980600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-7148e39c-12cc-4994-a191-5b433e762b0a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 12 17:19:47.980600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:47.980600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:47.980600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:47.980600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:47.980600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:47.980600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:47.980600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:47.980600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:47.980600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:47.980600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:47.980600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:47.980600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:47.980600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:47.980600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:47.980600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:47.981503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:47.981503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:47.981503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:47.981503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:47.981503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:47.981503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:47.981503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:47.981503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:47.981503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:47.981503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:47.981503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:47.981503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:47.981503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:47.981503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:47.981503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:47.981503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:47.982296 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:47.982296 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:47.982296 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:47.982296 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:47.982296 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:47.982296 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:47.982296 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:47.982296 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:47.982296 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:47.982296 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:47.982296 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 12 17:19:47.982296 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:19:47.982296 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 221/438] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:47 2022] GET /identity/v3/roles/member => generated 84 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:19:47.983237 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fd2f46-8e53-4a6d-8ee2-8647b354bd8f None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:47.983305 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fd2f46-8e53-4a6d-8ee2-8647b354bd8f None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:47.983603 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fd2f46-8e53-4a6d-8ee2-8647b354bd8f None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:47.984825 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 222/439] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:47 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:47.988286 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08fd2f46-8e53-4a6d-8ee2-8647b354bd8f None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:48.047153 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:48.067462 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 218/440] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:19:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 12 17:19:48.069116 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b15e191-988d-46fb-8ec1-efa94bc5b996 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:48.069409 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b15e191-988d-46fb-8ec1-efa94bc5b996 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:48.069687 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b15e191-988d-46fb-8ec1-efa94bc5b996 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:48.085546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-9b15e191-988d-46fb-8ec1-efa94bc5b996 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:48.086793 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-9b15e191-988d-46fb-8ec1-efa94bc5b996 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:48.093575 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08fd2f46-8e53-4a6d-8ee2-8647b354bd8f None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:48.122205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:48.127573 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1454f75c-1a31-4f28-bc29-2cbd75f5b653 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:48.128410 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1454f75c-1a31-4f28-bc29-2cbd75f5b653 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:48.133149 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1454f75c-1a31-4f28-bc29-2cbd75f5b653 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:48.133149 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1454f75c-1a31-4f28-bc29-2cbd75f5b653 None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:48.133149 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1454f75c-1a31-4f28-bc29-2cbd75f5b653 None admin] RBAC: Adding query filter params (name=member) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:48.133149 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1454f75c-1a31-4f28-bc29-2cbd75f5b653 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:48.133149 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1454f75c-1a31-4f28-bc29-2cbd75f5b653 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:48.141038 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 223/441] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Jul 12 17:19:47 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:48.141038 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1454f75c-1a31-4f28-bc29-2cbd75f5b653 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:48.166474 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 219/442] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:48.166474 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b15e191-988d-46fb-8ec1-efa94bc5b996 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:48.246657 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1454f75c-1a31-4f28-bc29-2cbd75f5b653 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:48.264019 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b15e191-988d-46fb-8ec1-efa94bc5b996 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:48.264255 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:48.285448 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85703d39-5234-41b4-8e31-703ae4298599 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:48.286308 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85703d39-5234-41b4-8e31-703ae4298599 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:48.286624 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85703d39-5234-41b4-8e31-703ae4298599 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:48.286908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85703d39-5234-41b4-8e31-703ae4298599 None admin] PATH_INFO: `/v3/projects/31b9ba63c2994be69d5ff224d61a9c25` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:48.288103 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85703d39-5234-41b4-8e31-703ae4298599 None admin] RBAC: Authorizing `identity:get_project(project_id=31b9ba63c2994be69d5ff224d61a9c25)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:48.289551 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85703d39-5234-41b4-8e31-703ae4298599 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:48.291469 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 224/443] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:48 2022] GET /identity/v3/projects/31b9ba63c2994be69d5ff224d61a9c25 => generated 302 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:48.295262 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85703d39-5234-41b4-8e31-703ae4298599 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:48.299456 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d68332d-6de4-4629-8fbd-405cfbbef8bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:48.300287 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d68332d-6de4-4629-8fbd-405cfbbef8bf None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:48.300746 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d68332d-6de4-4629-8fbd-405cfbbef8bf None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:48.301252 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d68332d-6de4-4629-8fbd-405cfbbef8bf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:48.341948 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:48.371767 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d68332d-6de4-4629-8fbd-405cfbbef8bf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:48.373257 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d68332d-6de4-4629-8fbd-405cfbbef8bf None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:48.397721 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85703d39-5234-41b4-8e31-703ae4298599 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:48.432695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308ea403-8e27-44b1-8079-e3a6e844611a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:48.433491 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308ea403-8e27-44b1-8079-e3a6e844611a None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:48.433901 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308ea403-8e27-44b1-8079-e3a6e844611a None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:48.434187 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308ea403-8e27-44b1-8079-e3a6e844611a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:48.444972 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-308ea403-8e27-44b1-8079-e3a6e844611a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:48.452804 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-308ea403-8e27-44b1-8079-e3a6e844611a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:19:48.452804 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:48.452804 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:48.452804 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:48.452804 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:48.452804 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:48.452804 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:48.452804 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:48.452804 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:48.452804 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:48.452804 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:48.452804 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:48.452804 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:48.452804 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:48.452804 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:48.452804 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:48.453715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:48.453715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:48.453715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:48.453715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:48.453715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:48.453715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:48.453715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:48.453715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:48.453715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:48.453715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:48.453715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:48.453715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:48.453715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:48.453715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:48.453715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:48.453715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:48.454571 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:48.454571 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:48.454571 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:48.454571 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:48.454571 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:48.454571 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:48.454571 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:48.454571 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:48.454571 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:48.454571 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:48.454571 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:19:48.454571 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:19:48.454571 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 225/444] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Jul 12 17:19:48 2022] GET /identity/v3/roles/admin => generated 83 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:48.455970 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308ea403-8e27-44b1-8079-e3a6e844611a None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:48.479850 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 220/445] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:19:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 12 17:19:48.481389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d68332d-6de4-4629-8fbd-405cfbbef8bf None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:48.505915 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:48.550444 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:48.559496 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308ea403-8e27-44b1-8079-e3a6e844611a None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:48.582486 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d68332d-6de4-4629-8fbd-405cfbbef8bf None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:48.597143 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f87624-5beb-4d3e-8bd6-c9a773e7fe94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:48.598001 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f87624-5beb-4d3e-8bd6-c9a773e7fe94 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:48.598287 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f87624-5beb-4d3e-8bd6-c9a773e7fe94 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:48.598595 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f87624-5beb-4d3e-8bd6-c9a773e7fe94 None admin] PATH_INFO: `/v3/groups/92246a55b7fd43469a1df54e34c8099f` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:48.599815 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f87624-5beb-4d3e-8bd6-c9a773e7fe94 None admin] RBAC: Authorizing `identity:get_group(group_id=92246a55b7fd43469a1df54e34c8099f)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:48.601245 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f87624-5beb-4d3e-8bd6-c9a773e7fe94 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:48.602904 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 226/446] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Jul 12 17:19:48 2022] GET /identity/v3/groups/92246a55b7fd43469a1df54e34c8099f => generated 226 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:48.603666 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f87624-5beb-4d3e-8bd6-c9a773e7fe94 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:48.617662 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2a50e3-5a93-4038-9a4b-53420f1e03b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:48.618335 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2a50e3-5a93-4038-9a4b-53420f1e03b4 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:48.618680 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2a50e3-5a93-4038-9a4b-53420f1e03b4 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:48.618962 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2a50e3-5a93-4038-9a4b-53420f1e03b4 None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:48.619710 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa2a50e3-5a93-4038-9a4b-53420f1e03b4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:48.620018 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa2a50e3-5a93-4038-9a4b-53420f1e03b4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:48.621411 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa2a50e3-5a93-4038-9a4b-53420f1e03b4 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:48.630218 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 221/447] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:48 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:48.631068 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2a50e3-5a93-4038-9a4b-53420f1e03b4 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:48.706896 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f87624-5beb-4d3e-8bd6-c9a773e7fe94 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:48.720675 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:48.733058 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2a50e3-5a93-4038-9a4b-53420f1e03b4 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:48.741033 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9895087a-4a57-4d3a-9a90-a8b1c39da25f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:48.741899 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9895087a-4a57-4d3a-9a90-a8b1c39da25f None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:48.742185 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9895087a-4a57-4d3a-9a90-a8b1c39da25f None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:48.742490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9895087a-4a57-4d3a-9a90-a8b1c39da25f None admin] PATH_INFO: `/v3/projects/31b9ba63c2994be69d5ff224d61a9c25` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:48.743751 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9895087a-4a57-4d3a-9a90-a8b1c39da25f None admin] RBAC: Authorizing `identity:get_project(project_id=31b9ba63c2994be69d5ff224d61a9c25)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:48.745183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9895087a-4a57-4d3a-9a90-a8b1c39da25f None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:48.746925 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 227/448] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:48 2022] GET /identity/v3/projects/31b9ba63c2994be69d5ff224d61a9c25 => generated 302 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:48.747678 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9895087a-4a57-4d3a-9a90-a8b1c39da25f None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:48.763815 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:48.768684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b80f316-8f50-451d-bb17-02eb6f4b6e89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:48.769344 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b80f316-8f50-451d-bb17-02eb6f4b6e89 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:48.769632 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b80f316-8f50-451d-bb17-02eb6f4b6e89 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:48.769921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b80f316-8f50-451d-bb17-02eb6f4b6e89 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:48.770791 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b80f316-8f50-451d-bb17-02eb6f4b6e89 None admin] RBAC: Adding query filter params (group.id=92246a55b7fd43469a1df54e34c8099f, role.id=0c239663092d459e8c9724963e78da76, scope.project.id=31b9ba63c2994be69d5ff224d61a9c25) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:48.771159 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b80f316-8f50-451d-bb17-02eb6f4b6e89 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:48.772606 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b80f316-8f50-451d-bb17-02eb6f4b6e89 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:48.778490 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 222/449] 10.222.0.16 () {56 vars in 1376 bytes} [Tue Jul 12 17:19:48 2022] GET /identity/v3/role_assignments?group.id=92246a55b7fd43469a1df54e34c8099f&scope.project.id=31b9ba63c2994be69d5ff224d61a9c25&role.id=0c239663092d459e8c9724963e78da76 => generated 611 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:48.779272 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b80f316-8f50-451d-bb17-02eb6f4b6e89 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:48.866373 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9895087a-4a57-4d3a-9a90-a8b1c39da25f None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:48.897899 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b80f316-8f50-451d-bb17-02eb6f4b6e89 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:48.904078 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76c7e9e-b2b0-430d-904f-b963d3e2bb26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:48.904736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76c7e9e-b2b0-430d-904f-b963d3e2bb26 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:48.905022 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76c7e9e-b2b0-430d-904f-b963d3e2bb26 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:48.905302 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76c7e9e-b2b0-430d-904f-b963d3e2bb26 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:48.943157 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5ec92e-e8ce-4c49-b1a4-74c413cccda7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:48.944841 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5ec92e-e8ce-4c49-b1a4-74c413cccda7 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:48.945132 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5ec92e-e8ce-4c49-b1a4-74c413cccda7 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:48.945417 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5ec92e-e8ce-4c49-b1a4-74c413cccda7 None admin] PATH_INFO: `/v3/groups/92246a55b7fd43469a1df54e34c8099f` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:48.946718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca5ec92e-e8ce-4c49-b1a4-74c413cccda7 None admin] RBAC: Authorizing `identity:get_group(group_id=92246a55b7fd43469a1df54e34c8099f)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:48.948624 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca5ec92e-e8ce-4c49-b1a4-74c413cccda7 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:48.950858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 223/450] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Jul 12 17:19:48 2022] GET /identity/v3/groups/92246a55b7fd43469a1df54e34c8099f => generated 226 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:48.956598 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5ec92e-e8ce-4c49-b1a4-74c413cccda7 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:48.964506 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:48.983464 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76c7e9e-b2b0-430d-904f-b963d3e2bb26 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:48.984962 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76c7e9e-b2b0-430d-904f-b963d3e2bb26 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:49.026846 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:49.066637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5ec92e-e8ce-4c49-b1a4-74c413cccda7 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:49.086512 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 228/451] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:19:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 12 17:19:49.092149 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76c7e9e-b2b0-430d-904f-b963d3e2bb26 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:49.100765 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dabe997f-a7ee-4e38-8a5b-46ecea63d55c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:49.101434 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabe997f-a7ee-4e38-8a5b-46ecea63d55c None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:49.101722 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabe997f-a7ee-4e38-8a5b-46ecea63d55c None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:49.102402 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dabe997f-a7ee-4e38-8a5b-46ecea63d55c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:49.103296 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dabe997f-a7ee-4e38-8a5b-46ecea63d55c None admin] RBAC: Adding query filter params (group.id=92246a55b7fd43469a1df54e34c8099f, role.id=4c5d52f873f94dc1b10918068376add5, scope.project.id=31b9ba63c2994be69d5ff224d61a9c25) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:49.103935 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dabe997f-a7ee-4e38-8a5b-46ecea63d55c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:49.105640 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dabe997f-a7ee-4e38-8a5b-46ecea63d55c None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:49.112258 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 224/452] 10.222.0.16 () {56 vars in 1376 bytes} [Tue Jul 12 17:19:48 2022] GET /identity/v3/role_assignments?group.id=92246a55b7fd43469a1df54e34c8099f&scope.project.id=31b9ba63c2994be69d5ff224d61a9c25&role.id=4c5d52f873f94dc1b10918068376add5 => generated 611 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:19:49.178031 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:49.197256 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76c7e9e-b2b0-430d-904f-b963d3e2bb26 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:49.234954 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54950e6-6bd7-4e82-a985-eafb69734be0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:49.236483 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54950e6-6bd7-4e82-a985-eafb69734be0 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:49.236945 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54950e6-6bd7-4e82-a985-eafb69734be0 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:49.237251 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54950e6-6bd7-4e82-a985-eafb69734be0 None admin] PATH_INFO: `/v3/projects/1d1f69915b004b66af4c48b7c59a15d8` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:49.238544 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54950e6-6bd7-4e82-a985-eafb69734be0 None admin] RBAC: Authorizing `identity:get_project(project_id=1d1f69915b004b66af4c48b7c59a15d8)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:49.240023 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f54950e6-6bd7-4e82-a985-eafb69734be0 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:49.241828 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 229/453] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:19:49 2022] GET /identity/v3/projects/1d1f69915b004b66af4c48b7c59a15d8 => generated 344 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:19:49.247686 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dabe997f-a7ee-4e38-8a5b-46ecea63d55c None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:49.355668 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dabe997f-a7ee-4e38-8a5b-46ecea63d55c None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:49.382990 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:49.390567 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4baf02c2-d64d-41bb-bb05-65829adb4058 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:49.391424 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4baf02c2-d64d-41bb-bb05-65829adb4058 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:49.391707 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4baf02c2-d64d-41bb-bb05-65829adb4058 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:49.395741 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4baf02c2-d64d-41bb-bb05-65829adb4058 None admin] PATH_INFO: `/v3/groups/059e96df72874559ab13e17f5c9d5cfc` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:49.395741 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4baf02c2-d64d-41bb-bb05-65829adb4058 None admin] RBAC: Authorizing `identity:get_group(group_id=059e96df72874559ab13e17f5c9d5cfc)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:49.395741 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4baf02c2-d64d-41bb-bb05-65829adb4058 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:49.396989 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 225/454] 10.222.0.16 () {56 vars in 1180 bytes} [Tue Jul 12 17:19:49 2022] GET /identity/v3/groups/059e96df72874559ab13e17f5c9d5cfc => generated 229 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:49.402993 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54950e6-6bd7-4e82-a985-eafb69734be0 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:49.507909 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54950e6-6bd7-4e82-a985-eafb69734be0 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:49.541844 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a58e34-3767-4fc1-9112-277be5d33f60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:49.542560 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a58e34-3767-4fc1-9112-277be5d33f60 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:49.542846 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a58e34-3767-4fc1-9112-277be5d33f60 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:49.543124 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a58e34-3767-4fc1-9112-277be5d33f60 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:49.543952 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a58e34-3767-4fc1-9112-277be5d33f60 None admin] RBAC: Adding query filter params (group.id=059e96df72874559ab13e17f5c9d5cfc, role.id=eb05bf286c894ea9bf44a6c023a4323b, scope.project.id=1d1f69915b004b66af4c48b7c59a15d8) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:49.544265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a58e34-3767-4fc1-9112-277be5d33f60 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:49.545693 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a58e34-3767-4fc1-9112-277be5d33f60 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:49.552216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 230/455] 10.222.0.16 () {56 vars in 1376 bytes} [Tue Jul 12 17:19:49 2022] GET /identity/v3/role_assignments?group.id=059e96df72874559ab13e17f5c9d5cfc&scope.project.id=1d1f69915b004b66af4c48b7c59a15d8&role.id=eb05bf286c894ea9bf44a6c023a4323b => generated 611 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:52.178588 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc2fbce-b628-4519-9f59-6e82e6623428 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:52.178588 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc2fbce-b628-4519-9f59-6e82e6623428 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:52.178588 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc2fbce-b628-4519-9f59-6e82e6623428 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:52.179706 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 226/456] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:52.183895 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6aceaa-b4cf-4c4c-8791-e012b6d1e02f None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:52.184287 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6aceaa-b4cf-4c4c-8791-e012b6d1e02f None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:52.184453 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6aceaa-b4cf-4c4c-8791-e012b6d1e02f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:52.207146 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-ea6aceaa-b4cf-4c4c-8791-e012b6d1e02f None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:52.208360 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-ea6aceaa-b4cf-4c4c-8791-e012b6d1e02f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:52.224634 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:52.253869 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582746b3-67d5-4328-b329-424d375c06fc None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:52.254518 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582746b3-67d5-4328-b329-424d375c06fc None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:52.255007 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582746b3-67d5-4328-b329-424d375c06fc None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:52.256814 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 227/457] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:52.262168 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf0630e-00b1-45af-a2c4-6554e7dcb212 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:52.262739 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf0630e-00b1-45af-a2c4-6554e7dcb212 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:52.263234 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf0630e-00b1-45af-a2c4-6554e7dcb212 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:52.291760 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-3cf0630e-00b1-45af-a2c4-6554e7dcb212 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:52.293123 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-3cf0630e-00b1-45af-a2c4-6554e7dcb212 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:52.301414 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 231/458] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:52.324695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea6aceaa-b4cf-4c4c-8791-e012b6d1e02f None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:52.430738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 228/459] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 12 17:19:52.485605 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea6aceaa-b4cf-4c4c-8791-e012b6d1e02f None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:52.488509 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf0630e-00b1-45af-a2c4-6554e7dcb212 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:52.539568 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0db2bf-7e49-44e1-9e43-8195ff0d9888 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:52.540370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0db2bf-7e49-44e1-9e43-8195ff0d9888 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:52.540655 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0db2bf-7e49-44e1-9e43-8195ff0d9888 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:52.540935 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0db2bf-7e49-44e1-9e43-8195ff0d9888 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:52.554905 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0db2bf-7e49-44e1-9e43-8195ff0d9888 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:52.556654 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0db2bf-7e49-44e1-9e43-8195ff0d9888 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:52.560611 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:52.580489 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-3f0db2bf-7e49-44e1-9e43-8195ff0d9888 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:19:52.580489 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:52.580489 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:52.580489 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:52.580489 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:52.580489 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:52.580489 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:52.580489 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:52.580489 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:52.580489 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:52.580489 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:52.580489 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:52.580489 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:19:52.580489 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:19:52.580489 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:19:52.581403 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:19:52.581403 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:52.581403 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:52.581403 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:52.581403 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:52.581403 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:52.581403 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:52.581403 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:52.581403 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:52.581403 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:52.581403 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:52.581403 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:52.581403 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:52.581403 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:52.581403 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:52.581403 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:52.581403 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:52.582260 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:19:52.582260 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:19:52.582260 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:19:52.582260 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:19:52.584289 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 232/460] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:19:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:19:52.585236 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f72c11b-09c6-412f-b01d-0aeb0fd392f1 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:52.585532 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f72c11b-09c6-412f-b01d-0aeb0fd392f1 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:52.585815 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f72c11b-09c6-412f-b01d-0aeb0fd392f1 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:52.587448 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 233/461] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:52.589483 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f72c11b-09c6-412f-b01d-0aeb0fd392f1 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:52.626523 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf0630e-00b1-45af-a2c4-6554e7dcb212 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:52.660950 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:52.670143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa447c91-327c-46fc-b695-d35594e43a5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:52.671164 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa447c91-327c-46fc-b695-d35594e43a5b None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:52.671600 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa447c91-327c-46fc-b695-d35594e43a5b None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:52.672583 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa447c91-327c-46fc-b695-d35594e43a5b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:52.686851 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa447c91-327c-46fc-b695-d35594e43a5b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:52.688519 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa447c91-327c-46fc-b695-d35594e43a5b None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:52.710167 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f72c11b-09c6-412f-b01d-0aeb0fd392f1 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:52.710963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-aa447c91-327c-46fc-b695-d35594e43a5b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:19:52.710963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:52.710963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:52.710963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:52.710963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:52.710963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:52.710963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:52.710963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:52.710963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:52.710963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:52.710963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:52.710963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:52.710963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:19:52.710963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:19:52.710963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:19:52.712006 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:19:52.712006 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:52.712006 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:52.712006 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:52.712006 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:52.712006 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:52.712006 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:52.712006 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:52.712006 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:52.712006 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:52.712006 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:52.712006 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:52.712006 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:52.712006 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:52.712006 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:52.712006 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:52.712006 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:52.713153 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:19:52.713153 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:19:52.713153 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:19:52.713153 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:52.715106 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 229/462] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:19:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:52.716459 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd7d113-6bbb-4eef-8626-02c37cfd0945 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:52.716877 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd7d113-6bbb-4eef-8626-02c37cfd0945 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:52.717304 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd7d113-6bbb-4eef-8626-02c37cfd0945 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:52.738601 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-8dd7d113-6bbb-4eef-8626-02c37cfd0945 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:52.740157 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-8dd7d113-6bbb-4eef-8626-02c37cfd0945 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:52.749120 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7e5c8d-6ffb-4380-a050-0e0e496d5e6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:52.749790 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7e5c8d-6ffb-4380-a050-0e0e496d5e6d None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:52.750073 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7e5c8d-6ffb-4380-a050-0e0e496d5e6d None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:52.750352 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7e5c8d-6ffb-4380-a050-0e0e496d5e6d None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:52.751774 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7e5c8d-6ffb-4380-a050-0e0e496d5e6d None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:52.752335 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7e5c8d-6ffb-4380-a050-0e0e496d5e6d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:52.754185 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7e5c8d-6ffb-4380-a050-0e0e496d5e6d None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:52.774259 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 234/463] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:19:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:52.775400 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7e5c8d-6ffb-4380-a050-0e0e496d5e6d None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:52.822545 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 230/464] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:52.823647 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd7d113-6bbb-4eef-8626-02c37cfd0945 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:52.838489 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:52.886926 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7e5c8d-6ffb-4380-a050-0e0e496d5e6d None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:52.924323 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40df4da-5879-41d0-bbc8-29a9d509a4a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:52.925279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40df4da-5879-41d0-bbc8-29a9d509a4a3 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:52.925937 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40df4da-5879-41d0-bbc8-29a9d509a4a3 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:52.926357 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40df4da-5879-41d0-bbc8-29a9d509a4a3 None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:52.927283 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f40df4da-5879-41d0-bbc8-29a9d509a4a3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:52.927745 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f40df4da-5879-41d0-bbc8-29a9d509a4a3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:52.929501 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f40df4da-5879-41d0-bbc8-29a9d509a4a3 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:52.943250 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 235/465] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:19:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:52.944539 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40df4da-5879-41d0-bbc8-29a9d509a4a3 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:52.949232 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd7d113-6bbb-4eef-8626-02c37cfd0945 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:52.960057 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:52.987758 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d281e00-027f-42f5-8e95-9ff3bd03a7b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:52.988577 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d281e00-027f-42f5-8e95-9ff3bd03a7b3 None admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:52.989388 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d281e00-027f-42f5-8e95-9ff3bd03a7b3 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:52.989910 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d281e00-027f-42f5-8e95-9ff3bd03a7b3 None admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:52.991086 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d281e00-027f-42f5-8e95-9ff3bd03a7b3 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:52.993193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d281e00-027f-42f5-8e95-9ff3bd03a7b3 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:53.057336 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40df4da-5879-41d0-bbc8-29a9d509a4a3 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:53.074925 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:53.092698 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a307736b-d51f-452a-9ca4-d0db1954ee69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:53.093489 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a307736b-d51f-452a-9ca4-d0db1954ee69 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:53.093818 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a307736b-d51f-452a-9ca4-d0db1954ee69 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:53.094132 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a307736b-d51f-452a-9ca4-d0db1954ee69 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:53.105001 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a307736b-d51f-452a-9ca4-d0db1954ee69 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:53.106571 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a307736b-d51f-452a-9ca4-d0db1954ee69 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:53.123413 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-a307736b-d51f-452a-9ca4-d0db1954ee69 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:19:53.123413 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:53.123413 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:53.123413 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:53.123413 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:53.123413 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:53.123413 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:53.123413 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:53.123413 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:53.123413 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:53.123413 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:53.123413 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:53.123413 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:19:53.123413 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:19:53.123413 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:19:53.124494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:19:53.124494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:53.124494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:53.124494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:53.124494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:53.124494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:53.124494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:53.124494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:53.124494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:53.124494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:53.124494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:53.124494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:53.124494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:53.124494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:53.124494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:53.124494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:53.124494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:53.125650 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:19:53.125650 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:19:53.125650 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:19:53.125650 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:19:53.125650 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 236/466] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:19:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:53.126558 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a307736b-d51f-452a-9ca4-d0db1954ee69 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:53.131157 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 231/467] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Jul 12 17:19:52 2022] POST /identity/v3/users => generated 246 bytes in 308 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:19:53.132672 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a0ec20-f4c0-4796-94d8-c2a016e14bc7 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:53.133143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a0ec20-f4c0-4796-94d8-c2a016e14bc7 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:53.133559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a0ec20-f4c0-4796-94d8-c2a016e14bc7 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:53.139747 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 232/468] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:53 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:53.139747 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a0ec20-f4c0-4796-94d8-c2a016e14bc7 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:53.264863 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a307736b-d51f-452a-9ca4-d0db1954ee69 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:53.300640 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a0ec20-f4c0-4796-94d8-c2a016e14bc7 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:53.308785 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab956957-817f-4e84-82c6-d536bc27e50f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:53.309468 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab956957-817f-4e84-82c6-d536bc27e50f None admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:53.309763 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab956957-817f-4e84-82c6-d536bc27e50f None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:53.310049 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab956957-817f-4e84-82c6-d536bc27e50f None admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:53.310826 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab956957-817f-4e84-82c6-d536bc27e50f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:53.312574 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab956957-817f-4e84-82c6-d536bc27e50f None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:53.342935 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5c446e-693e-443b-96fb-04fbb5c2c313 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:53.343607 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5c446e-693e-443b-96fb-04fbb5c2c313 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:53.343918 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5c446e-693e-443b-96fb-04fbb5c2c313 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:53.344203 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5c446e-693e-443b-96fb-04fbb5c2c313 None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:53.345512 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5c446e-693e-443b-96fb-04fbb5c2c313 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:53.345832 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5c446e-693e-443b-96fb-04fbb5c2c313 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:53.347345 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5c446e-693e-443b-96fb-04fbb5c2c313 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:53.362097 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 233/469] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:19:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:53.363789 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d98c52-547b-4abe-b10b-0a8172bee98a None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:53.364090 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d98c52-547b-4abe-b10b-0a8172bee98a None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:53.364386 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d98c52-547b-4abe-b10b-0a8172bee98a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:53.382120 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-28d98c52-547b-4abe-b10b-0a8172bee98a None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:53.383373 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-28d98c52-547b-4abe-b10b-0a8172bee98a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:53.402807 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:53.404825 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 237/470] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Jul 12 17:19:53 2022] POST /identity/v3/users => generated 248 bytes in 279 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:19:53.406749 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab956957-817f-4e84-82c6-d536bc27e50f None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:53.446026 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:53.476707 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 234/471] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:53.504073 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d98c52-547b-4abe-b10b-0a8172bee98a None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:53.548736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab956957-817f-4e84-82c6-d536bc27e50f None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:53.624173 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d98c52-547b-4abe-b10b-0a8172bee98a None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:53.629450 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-644c3579-461e-41d6-99ce-fa01dd6e8d02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:53.630149 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644c3579-461e-41d6-99ce-fa01dd6e8d02 None admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:53.630463 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644c3579-461e-41d6-99ce-fa01dd6e8d02 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:53.630760 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644c3579-461e-41d6-99ce-fa01dd6e8d02 None admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:53.631492 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-644c3579-461e-41d6-99ce-fa01dd6e8d02 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:53.632969 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-644c3579-461e-41d6-99ce-fa01dd6e8d02 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:53.668697 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b450026-fa69-4145-9ec4-eb5a0a397f3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:53.669516 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b450026-fa69-4145-9ec4-eb5a0a397f3b None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:53.670470 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b450026-fa69-4145-9ec4-eb5a0a397f3b None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:53.670470 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b450026-fa69-4145-9ec4-eb5a0a397f3b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:53.681603 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b450026-fa69-4145-9ec4-eb5a0a397f3b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:53.683728 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b450026-fa69-4145-9ec4-eb5a0a397f3b None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:53.702010 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-9b450026-fa69-4145-9ec4-eb5a0a397f3b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:19:53.702010 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:53.702010 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:53.702010 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:53.702010 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:53.702010 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:53.702010 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:53.702010 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:53.702010 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:53.702010 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:53.702010 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:53.702010 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:53.702010 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:19:53.702010 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:19:53.702010 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:19:53.702998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:19:53.702998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:53.702998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:53.702998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:53.702998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:53.702998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:53.702998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:53.702998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:53.702998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:53.702998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:53.702998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:53.702998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:53.702998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:53.702998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:53.702998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:53.702998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:53.702998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:53.703914 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:19:53.703914 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:19:53.703914 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:19:53.703914 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:53.703914 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 235/472] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:19:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:53.706024 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35a2112-3144-42ab-8bd5-2bbfb06da3ea None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:53.706024 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35a2112-3144-42ab-8bd5-2bbfb06da3ea None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:53.706292 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35a2112-3144-42ab-8bd5-2bbfb06da3ea None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:53.707730 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 236/473] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:53.711072 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35a2112-3144-42ab-8bd5-2bbfb06da3ea None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:53.720740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:53.747120 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 238/474] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Jul 12 17:19:53 2022] POST /identity/v3/users => generated 248 bytes in 341 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:19:53.748792 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace70b27-7042-4801-bdda-ed5b8d95ec17 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:53.749286 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace70b27-7042-4801-bdda-ed5b8d95ec17 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:53.749791 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace70b27-7042-4801-bdda-ed5b8d95ec17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:53.799695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-ace70b27-7042-4801-bdda-ed5b8d95ec17 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:53.802379 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-ace70b27-7042-4801-bdda-ed5b8d95ec17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:53.875200 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35a2112-3144-42ab-8bd5-2bbfb06da3ea None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:53.919179 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8227e811-b4b2-4ca1-b3e6-bc864b84c3e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:53.919531 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8227e811-b4b2-4ca1-b3e6-bc864b84c3e7 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:53.919847 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8227e811-b4b2-4ca1-b3e6-bc864b84c3e7 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:53.920134 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8227e811-b4b2-4ca1-b3e6-bc864b84c3e7 None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:53.920919 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8227e811-b4b2-4ca1-b3e6-bc864b84c3e7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:53.921239 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8227e811-b4b2-4ca1-b3e6-bc864b84c3e7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:53.922778 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8227e811-b4b2-4ca1-b3e6-bc864b84c3e7 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:53.938075 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 237/475] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:19:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:19:53.943973 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8227e811-b4b2-4ca1-b3e6-bc864b84c3e7 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:53.952007 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 239/476] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:53.977414 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace70b27-7042-4801-bdda-ed5b8d95ec17 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:53.991240 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:54.035333 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:54.076583 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8227e811-b4b2-4ca1-b3e6-bc864b84c3e7 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:54.122363 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace70b27-7042-4801-bdda-ed5b8d95ec17 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:54.127870 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-163066a4-3090-42b5-a1e2-63dfc38a6c87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:54.128830 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163066a4-3090-42b5-a1e2-63dfc38a6c87 None admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:54.129409 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163066a4-3090-42b5-a1e2-63dfc38a6c87 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:54.129872 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163066a4-3090-42b5-a1e2-63dfc38a6c87 None admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:54.130799 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-163066a4-3090-42b5-a1e2-63dfc38a6c87 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:54.133100 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-163066a4-3090-42b5-a1e2-63dfc38a6c87 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:54.167494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9b694d-0e4e-4bfb-af2e-f0457f539747 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:54.168269 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9b694d-0e4e-4bfb-af2e-f0457f539747 None admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:54.168626 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9b694d-0e4e-4bfb-af2e-f0457f539747 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:54.168970 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9b694d-0e4e-4bfb-af2e-f0457f539747 None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:54.169822 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9b694d-0e4e-4bfb-af2e-f0457f539747 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:54.171380 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9b694d-0e4e-4bfb-af2e-f0457f539747 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:54.226304 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.sql.core [None req-ef9b694d-0e4e-4bfb-af2e-f0457f539747 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Jul 12 17:19:54.226304 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Jul 12 17:19:54.226304 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [parameters: {'id': '62e7f973691d48feb20349b57d5f699d', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Jul 12 17:19:54.226304 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=80452) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Jul 12 17:19:54.234544 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 238/477] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Jul 12 17:19:53 2022] POST /identity/v3/users => generated 249 bytes in 291 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:19:54.246531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-ef9b694d-0e4e-4bfb-af2e-f0457f539747 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Jul 12 17:19:54.246531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:54.246531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 12 17:19:54.246531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 12 17:19:54.246531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 12 17:19:54.246531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 12 17:19:54.246531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 12 17:19:54.246531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application result = self._query(query) Jul 12 17:19:54.246531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 12 17:19:54.246531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application conn.query(q) Jul 12 17:19:54.246531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 12 17:19:54.246531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 12 17:19:54.246531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 12 17:19:54.246531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application result.read() Jul 12 17:19:54.246531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 12 17:19:54.247508 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 12 17:19:54.247508 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 12 17:19:54.247508 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 12 17:19:54.247508 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 12 17:19:54.247508 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 12 17:19:54.247508 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 12 17:19:54.247508 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 12 17:19:54.247508 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Jul 12 17:19:54.247508 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:19:54.247508 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Jul 12 17:19:54.247508 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:19:54.247508 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:54.247508 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Jul 12 17:19:54.247508 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return method(*args, **kwargs) Jul 12 17:19:54.247508 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Jul 12 17:19:54.247508 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return ref.to_dict() Jul 12 17:19:54.247508 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 12 17:19:54.247508 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application next(self.gen) Jul 12 17:19:54.248442 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Jul 12 17:19:54.248442 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application yield resource Jul 12 17:19:54.248442 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 12 17:19:54.248442 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application next(self.gen) Jul 12 17:19:54.248442 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Jul 12 17:19:54.248442 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self.session.rollback() Jul 12 17:19:54.248442 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jul 12 17:19:54.248442 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self.force_reraise() Jul 12 17:19:54.248442 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jul 12 17:19:54.248442 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise self.value Jul 12 17:19:54.248442 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Jul 12 17:19:54.248442 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Jul 12 17:19:54.248442 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Jul 12 17:19:54.248442 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application session.commit() Jul 12 17:19:54.248442 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Jul 12 17:19:54.248442 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Jul 12 17:19:54.248442 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Jul 12 17:19:54.249300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self._prepare_impl() Jul 12 17:19:54.249300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Jul 12 17:19:54.249300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self.session.flush() Jul 12 17:19:54.249300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Jul 12 17:19:54.249300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self._flush(objects) Jul 12 17:19:54.249300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Jul 12 17:19:54.249300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Jul 12 17:19:54.249300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Jul 12 17:19:54.249300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application compat.raise_( Jul 12 17:19:54.249300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 12 17:19:54.249300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception Jul 12 17:19:54.249300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Jul 12 17:19:54.249300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application flush_context.execute() Jul 12 17:19:54.249300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Jul 12 17:19:54.249300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rec.execute(self) Jul 12 17:19:54.249300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Jul 12 17:19:54.249300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Jul 12 17:19:54.250166 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Jul 12 17:19:54.250166 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application _emit_insert_statements( Jul 12 17:19:54.250166 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Jul 12 17:19:54.250166 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application c = connection._execute_20( Jul 12 17:19:54.250166 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 Jul 12 17:19:54.250166 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Jul 12 17:19:54.250166 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 332, in _execute_on_connection Jul 12 17:19:54.250166 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Jul 12 17:19:54.250166 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement Jul 12 17:19:54.250166 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ret = self._execute_context( Jul 12 17:19:54.250166 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context Jul 12 17:19:54.250166 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Jul 12 17:19:54.250166 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2041, in _handle_dbapi_exception Jul 12 17:19:54.250166 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Jul 12 17:19:54.250166 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 12 17:19:54.250166 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception Jul 12 17:19:54.252738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 12 17:19:54.252738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 12 17:19:54.252738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 12 17:19:54.252738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 12 17:19:54.252738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 12 17:19:54.252738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application result = self._query(query) Jul 12 17:19:54.252738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 12 17:19:54.252738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application conn.query(q) Jul 12 17:19:54.252738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 12 17:19:54.252738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 12 17:19:54.252738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 12 17:19:54.252738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application result.read() Jul 12 17:19:54.252738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 12 17:19:54.252738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 12 17:19:54.252738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 12 17:19:54.252738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 12 17:19:54.253542 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 12 17:19:54.253542 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 12 17:19:54.253542 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 12 17:19:54.253542 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 12 17:19:54.253542 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Jul 12 17:19:54.253542 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Jul 12 17:19:54.253542 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [parameters: {'id': '62e7f973691d48feb20349b57d5f699d', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Jul 12 17:19:54.253542 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Jul 12 17:19:54.253542 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:19:54.253542 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 12 17:19:54.253542 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:19:54.253542 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:54.253542 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:54.253542 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:54.253542 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:54.253542 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:54.254370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:54.254370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:54.254370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:54.254370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:54.254370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:54.254370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:54.254370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Jul 12 17:19:54.254370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Jul 12 17:19:54.254370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:54.254370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:54.254370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Jul 12 17:19:54.254370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Jul 12 17:19:54.254370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Jul 12 17:19:54.254370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Jul 12 17:19:54.254370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Jul 12 17:19:54.254370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:19:54.256905 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 240/478] 10.222.0.16 () {60 vars in 1131 bytes} [Tue Jul 12 17:19:53 2022] POST /identity/v3/roles => generated 144 bytes in 271 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:19:54.263353 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-163066a4-3090-42b5-a1e2-63dfc38a6c87 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:54.404766 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-163066a4-3090-42b5-a1e2-63dfc38a6c87 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:54.431309 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:54.444415 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a13b160-f0db-40b4-b22c-2fb809d8310f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:54.445605 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a13b160-f0db-40b4-b22c-2fb809d8310f None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:54.446147 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a13b160-f0db-40b4-b22c-2fb809d8310f None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:54.446639 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a13b160-f0db-40b4-b22c-2fb809d8310f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:54.465369 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a13b160-f0db-40b4-b22c-2fb809d8310f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:54.474777 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-8a13b160-f0db-40b4-b22c-2fb809d8310f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 12 17:19:54.474777 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:54.474777 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:54.474777 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:54.474777 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:54.474777 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:54.474777 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:54.474777 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:54.474777 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:54.474777 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:54.474777 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:54.474777 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:54.474777 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:54.474777 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:54.474777 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:54.474777 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:54.475663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:54.475663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:54.475663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:54.475663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:54.475663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:54.475663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:54.475663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:54.475663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:54.475663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:54.475663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:54.475663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:54.475663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:54.475663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:54.475663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:54.475663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:54.475663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:54.476466 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:54.476466 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:54.476466 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:54.476466 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:54.476466 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:54.476466 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:54.476466 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:54.476466 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:54.476466 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:54.476466 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:54.476466 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 12 17:19:54.476466 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:54.479268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 239/479] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Jul 12 17:19:54 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:54.487529 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9b694d-0e4e-4bfb-af2e-f0457f539747 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:54.582061 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:54.611596 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9b694d-0e4e-4bfb-af2e-f0457f539747 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:54.652609 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e354675e-157d-4e07-bcfe-5d62ff45e865 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:54.653299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e354675e-157d-4e07-bcfe-5d62ff45e865 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:54.653588 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e354675e-157d-4e07-bcfe-5d62ff45e865 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:54.653868 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e354675e-157d-4e07-bcfe-5d62ff45e865 None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:54.654660 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e354675e-157d-4e07-bcfe-5d62ff45e865 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:54.655026 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e354675e-157d-4e07-bcfe-5d62ff45e865 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:54.656487 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e354675e-157d-4e07-bcfe-5d62ff45e865 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:54.667604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 241/480] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:19:54 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:55.668357 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fca193-cb5f-432a-9b4b-68308f2f9f19 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:55.669328 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fca193-cb5f-432a-9b4b-68308f2f9f19 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:55.669817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fca193-cb5f-432a-9b4b-68308f2f9f19 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:55.671634 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 240/481] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:55.678141 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099b5c2c-23db-4db7-bf59-0e3a1a36afa7 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:55.678598 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099b5c2c-23db-4db7-bf59-0e3a1a36afa7 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:55.678824 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-099b5c2c-23db-4db7-bf59-0e3a1a36afa7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:55.703460 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-099b5c2c-23db-4db7-bf59-0e3a1a36afa7 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:55.704896 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-099b5c2c-23db-4db7-bf59-0e3a1a36afa7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:55.796162 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 242/482] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:55.833393 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9fca193-cb5f-432a-9b4b-68308f2f9f19 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:55.943605 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7beadb20-e15b-4202-aebf-72fceaafc45c None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:55.944256 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7beadb20-e15b-4202-aebf-72fceaafc45c None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:55.944737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7beadb20-e15b-4202-aebf-72fceaafc45c None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:55.946817 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 243/483] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:55.952754 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2490eb0-8333-4bf7-a820-3df2ed6d38bf None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:55.953512 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2490eb0-8333-4bf7-a820-3df2ed6d38bf None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:55.954124 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2490eb0-8333-4bf7-a820-3df2ed6d38bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:55.968264 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9fca193-cb5f-432a-9b4b-68308f2f9f19 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:55.972779 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-f2490eb0-8333-4bf7-a820-3df2ed6d38bf None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:55.974216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-f2490eb0-8333-4bf7-a820-3df2ed6d38bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:56.016752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c09d1f-3e88-4151-a212-fd4194b34b5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:56.017583 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c09d1f-3e88-4151-a212-fd4194b34b5a None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:56.017872 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c09d1f-3e88-4151-a212-fd4194b34b5a None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:56.018138 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c09d1f-3e88-4151-a212-fd4194b34b5a None admin] PATH_INFO: `/v3/roles/service` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:56.038496 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c09d1f-3e88-4151-a212-fd4194b34b5a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:56.043619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-d0c09d1f-3e88-4151-a212-fd4194b34b5a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:19:56.043619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:56.043619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:56.043619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:56.043619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:56.043619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:56.043619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:56.043619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:56.043619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:56.043619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:56.043619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:56.043619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:56.043619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:56.043619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:56.043619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:56.043619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:56.044481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:56.044481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:56.044481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:56.044481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:56.044481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:56.044481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:56.044481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:56.044481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:56.044481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:56.044481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:56.044481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:56.044481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:56.044481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:56.044481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:56.044481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:56.044481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:56.045290 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:56.045290 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:56.045290 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:56.045290 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:56.045290 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:56.045290 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:56.045290 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:56.045290 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:56.045290 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:56.045290 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:56.045290 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:19:56.045290 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:56.045290 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 241/484] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Jul 12 17:19:55 2022] GET /identity/v3/roles/service => generated 85 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:56.055055 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c09d1f-3e88-4151-a212-fd4194b34b5a None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:56.086548 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:56.089484 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 244/485] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 12 17:19:56.114000 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:56.143380 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2490eb0-8333-4bf7-a820-3df2ed6d38bf None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:56.212078 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c09d1f-3e88-4151-a212-fd4194b34b5a None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:56.260772 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d1aabb-6a3a-4058-9618-5a041b2d8aad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:56.261813 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d1aabb-6a3a-4058-9618-5a041b2d8aad None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:56.263296 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d1aabb-6a3a-4058-9618-5a041b2d8aad None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:56.263775 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d1aabb-6a3a-4058-9618-5a041b2d8aad None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:56.264717 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d1aabb-6a3a-4058-9618-5a041b2d8aad None admin] RBAC: Adding query filter params (name=service) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:56.267688 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d1aabb-6a3a-4058-9618-5a041b2d8aad None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:56.269362 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d1aabb-6a3a-4058-9618-5a041b2d8aad None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:56.288783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 242/486] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Jul 12 17:19:56 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:56.294452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d1aabb-6a3a-4058-9618-5a041b2d8aad None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:56.322922 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2490eb0-8333-4bf7-a820-3df2ed6d38bf None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:56.361857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53236cea-e073-4e5b-b73b-a794da216a8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:56.365410 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53236cea-e073-4e5b-b73b-a794da216a8d None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:56.365909 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53236cea-e073-4e5b-b73b-a794da216a8d None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:56.366382 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53236cea-e073-4e5b-b73b-a794da216a8d None admin] PATH_INFO: `/v3/roles/service` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:56.378667 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53236cea-e073-4e5b-b73b-a794da216a8d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:56.386738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-53236cea-e073-4e5b-b73b-a794da216a8d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:19:56.386738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:56.386738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:56.386738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:56.386738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:56.386738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:56.386738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:56.386738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:56.386738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:56.386738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:56.386738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:56.386738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:56.386738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:56.386738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:56.386738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:56.386738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:56.387749 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:56.387749 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:56.387749 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:56.387749 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:56.387749 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:56.387749 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:56.387749 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:56.387749 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:56.387749 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:56.387749 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:56.387749 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:56.387749 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:56.387749 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:56.387749 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:56.387749 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:56.387749 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:56.388556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:56.388556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:56.388556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:56.388556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:56.388556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:56.388556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:56.388556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:56.388556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:56.388556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:56.388556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:56.388556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:19:56.388556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:19:56.391445 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 245/487] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Jul 12 17:19:56 2022] GET /identity/v3/roles/service => generated 85 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:56.394738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60efdcfa-cc0e-4166-9f41-bbac25599b6d None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:56.395259 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60efdcfa-cc0e-4166-9f41-bbac25599b6d None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:56.395733 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60efdcfa-cc0e-4166-9f41-bbac25599b6d None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:56.397577 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 246/488] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:56 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:56.398508 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60efdcfa-cc0e-4166-9f41-bbac25599b6d None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:56.416224 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d1aabb-6a3a-4058-9618-5a041b2d8aad None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:56.428105 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:56.453088 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7177299-0a10-47ec-9e72-a2ef713f4c1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:56.453996 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7177299-0a10-47ec-9e72-a2ef713f4c1c None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:56.455017 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7177299-0a10-47ec-9e72-a2ef713f4c1c None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:56.455443 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7177299-0a10-47ec-9e72-a2ef713f4c1c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:56.462189 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:56.466758 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7177299-0a10-47ec-9e72-a2ef713f4c1c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:56.468162 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7177299-0a10-47ec-9e72-a2ef713f4c1c None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:56.487858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-b7177299-0a10-47ec-9e72-a2ef713f4c1c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:19:56.487858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:56.487858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:56.487858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:56.487858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:56.487858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:56.487858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:56.487858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:56.487858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:56.487858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:56.487858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:56.487858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:56.487858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:19:56.487858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:19:56.487858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:19:56.488961 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:19:56.488961 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:56.488961 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:56.488961 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:56.488961 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:56.488961 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:56.488961 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:56.488961 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:56.488961 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:56.488961 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:56.488961 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:56.488961 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:56.488961 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:56.488961 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:56.488961 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:56.488961 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:56.488961 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:56.489827 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:19:56.489827 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:19:56.489827 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:19:56.489827 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:56.493475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 243/489] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:19:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:19:56.494824 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad833060-744c-47ea-8dd3-284c525838e1 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:56.495118 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad833060-744c-47ea-8dd3-284c525838e1 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:56.495400 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad833060-744c-47ea-8dd3-284c525838e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:56.512621 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-ad833060-744c-47ea-8dd3-284c525838e1 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:56.513823 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-ad833060-744c-47ea-8dd3-284c525838e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:56.519602 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60efdcfa-cc0e-4166-9f41-bbac25599b6d None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:56.557296 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec472d7f-aaba-46c9-9eec-13e850bae670 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:56.558237 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec472d7f-aaba-46c9-9eec-13e850bae670 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:56.558694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec472d7f-aaba-46c9-9eec-13e850bae670 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:56.559111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec472d7f-aaba-46c9-9eec-13e850bae670 None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:56.560046 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec472d7f-aaba-46c9-9eec-13e850bae670 None admin] RBAC: Adding query filter params (name=service) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:56.560521 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec472d7f-aaba-46c9-9eec-13e850bae670 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:56.562066 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec472d7f-aaba-46c9-9eec-13e850bae670 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:56.584593 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 247/490] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Jul 12 17:19:56 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:56.585736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec472d7f-aaba-46c9-9eec-13e850bae670 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:56.590884 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 244/491] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:56.593396 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad833060-744c-47ea-8dd3-284c525838e1 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:56.718852 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:56.719482 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec472d7f-aaba-46c9-9eec-13e850bae670 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:56.726844 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad833060-744c-47ea-8dd3-284c525838e1 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:56.745005 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:56.767112 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87386f54-4b2f-4b08-b7de-d6dd57b82509 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:56.767965 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87386f54-4b2f-4b08-b7de-d6dd57b82509 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:56.768258 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87386f54-4b2f-4b08-b7de-d6dd57b82509 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:56.768633 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87386f54-4b2f-4b08-b7de-d6dd57b82509 None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:56.769460 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87386f54-4b2f-4b08-b7de-d6dd57b82509 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:56.769814 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87386f54-4b2f-4b08-b7de-d6dd57b82509 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:56.771298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87386f54-4b2f-4b08-b7de-d6dd57b82509 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:56.771654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550c9956-0bc9-495a-a80a-31471cf0d166 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:56.772776 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550c9956-0bc9-495a-a80a-31471cf0d166 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:56.773220 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550c9956-0bc9-495a-a80a-31471cf0d166 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:56.774123 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550c9956-0bc9-495a-a80a-31471cf0d166 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:56.786522 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 248/492] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:19:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:56.787644 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87386f54-4b2f-4b08-b7de-d6dd57b82509 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:56.791416 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550c9956-0bc9-495a-a80a-31471cf0d166 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:56.793265 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550c9956-0bc9-495a-a80a-31471cf0d166 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:56.817591 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-550c9956-0bc9-495a-a80a-31471cf0d166 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:19:56.817591 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:56.817591 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:56.817591 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:56.817591 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:56.817591 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:56.817591 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:56.817591 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:56.817591 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:56.817591 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:56.817591 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:56.817591 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:56.817591 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:19:56.817591 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:19:56.817591 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:19:56.818525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:19:56.818525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:56.818525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:56.818525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:56.818525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:56.818525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:56.818525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:56.818525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:56.818525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:56.818525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:56.818525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:56.818525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:56.818525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:56.818525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:56.818525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:56.818525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:56.818525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:56.819457 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:19:56.819457 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:19:56.819457 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:19:56.819457 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:56.821740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 245/493] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:19:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:56.822823 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550c9956-0bc9-495a-a80a-31471cf0d166 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:56.928621 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87386f54-4b2f-4b08-b7de-d6dd57b82509 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:56.950167 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550c9956-0bc9-495a-a80a-31471cf0d166 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:56.977598 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f55151-1b03-4e6a-8011-c1d6df1bf8d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:56.978906 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f55151-1b03-4e6a-8011-c1d6df1bf8d9 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:56.980031 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f55151-1b03-4e6a-8011-c1d6df1bf8d9 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:56.980482 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f55151-1b03-4e6a-8011-c1d6df1bf8d9 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:56.991438 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430c260e-3c32-49c1-9a93-2f259c939db4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:56.992438 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430c260e-3c32-49c1-9a93-2f259c939db4 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:56.992766 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430c260e-3c32-49c1-9a93-2f259c939db4 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:56.993281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430c260e-3c32-49c1-9a93-2f259c939db4 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:56.999988 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98f55151-1b03-4e6a-8011-c1d6df1bf8d9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:57.001599 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430c260e-3c32-49c1-9a93-2f259c939db4 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:57.002990 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430c260e-3c32-49c1-9a93-2f259c939db4 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:57.004920 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:57.010032 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-98f55151-1b03-4e6a-8011-c1d6df1bf8d9 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:19:57.010032 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:57.010032 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:57.010032 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:57.010032 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:57.010032 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:57.010032 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:57.010032 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:57.010032 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:57.010032 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:57.010032 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:57.010032 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:57.010032 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:57.010032 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:57.010032 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:57.010032 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:57.011043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:57.011043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:57.011043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:57.011043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:57.011043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:57.011043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:57.011043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:57.011043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:57.011043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:57.011043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:57.011043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:57.011043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:57.011043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:57.011043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:57.011043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:57.011043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:57.011921 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:57.011921 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:57.011921 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:57.011921 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:57.011921 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:57.011921 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:57.011921 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:57.011921 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:57.011921 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:57.011921 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:57.011921 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:19:57.011921 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:19:57.012530 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 249/494] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Jul 12 17:19:56 2022] GET /identity/v3/roles/service => generated 85 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:57.013200 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f55151-1b03-4e6a-8011-c1d6df1bf8d9 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:57.019994 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-430c260e-3c32-49c1-9a93-2f259c939db4 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 12 17:19:57.019994 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:57.019994 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:57.019994 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:57.019994 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:57.019994 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:57.019994 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:57.019994 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:57.019994 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:57.019994 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:57.019994 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:57.019994 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:57.019994 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:19:57.019994 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:19:57.019994 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:19:57.019994 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:19:57.020939 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:57.020939 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:57.020939 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:19:57.020939 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:19:57.020939 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:19:57.020939 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:19:57.020939 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:57.020939 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:57.020939 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:57.020939 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:57.020939 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:57.020939 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:57.020939 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:57.020939 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:57.020939 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:57.020939 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:57.020939 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:57.020939 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:57.021838 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:57.021838 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:57.021838 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:19:57.021838 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:19:57.021838 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:19:57.021838 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:19:57.021838 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:19:57.021838 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:19:57.021838 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 12 17:19:57.021838 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:57.025757 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 246/495] 10.222.0.16 () {56 vars in 1093 bytes} [Tue Jul 12 17:19:56 2022] GET /identity/v3/users/nova => generated 82 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:57.027123 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb2c6c4-0faf-4915-aff4-7efb13d90e28 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:57.027456 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb2c6c4-0faf-4915-aff4-7efb13d90e28 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:57.027743 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb2c6c4-0faf-4915-aff4-7efb13d90e28 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:57.029222 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 247/496] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:57.029687 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb2c6c4-0faf-4915-aff4-7efb13d90e28 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:57.059524 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:57.122222 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f55151-1b03-4e6a-8011-c1d6df1bf8d9 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:57.145958 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb2c6c4-0faf-4915-aff4-7efb13d90e28 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:57.156746 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96df4427-1d97-4af9-8ac7-19b5399c6f51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:57.157932 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96df4427-1d97-4af9-8ac7-19b5399c6f51 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:57.158238 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96df4427-1d97-4af9-8ac7-19b5399c6f51 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:57.158613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96df4427-1d97-4af9-8ac7-19b5399c6f51 None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:57.159403 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96df4427-1d97-4af9-8ac7-19b5399c6f51 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:57.159672 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96df4427-1d97-4af9-8ac7-19b5399c6f51 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:57.161032 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96df4427-1d97-4af9-8ac7-19b5399c6f51 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:57.177876 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 250/497] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:19:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:57.177876 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96df4427-1d97-4af9-8ac7-19b5399c6f51 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:57.194525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116194db-48ef-4d94-922b-04e878af9cac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:57.195165 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116194db-48ef-4d94-922b-04e878af9cac None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:57.195469 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116194db-48ef-4d94-922b-04e878af9cac None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:57.195740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116194db-48ef-4d94-922b-04e878af9cac None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:57.196528 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-116194db-48ef-4d94-922b-04e878af9cac None admin] RBAC: Adding query filter params (name=service) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:57.196843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-116194db-48ef-4d94-922b-04e878af9cac None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:57.198246 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-116194db-48ef-4d94-922b-04e878af9cac None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:57.207192 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 248/498] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Jul 12 17:19:57 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:57.212699 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f13dfb5-85b4-4ee3-b61b-a0b364d7460c None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:57.213001 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f13dfb5-85b4-4ee3-b61b-a0b364d7460c None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:57.213287 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f13dfb5-85b4-4ee3-b61b-a0b364d7460c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:57.230846 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-2f13dfb5-85b4-4ee3-b61b-a0b364d7460c None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:57.231987 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-2f13dfb5-85b4-4ee3-b61b-a0b364d7460c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:57.281222 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:57.286961 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96df4427-1d97-4af9-8ac7-19b5399c6f51 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:57.291499 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:57.306266 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 249/499] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:57.307216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f13dfb5-85b4-4ee3-b61b-a0b364d7460c None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:57.341165 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819a4fee-5bf7-4dfd-847e-2c0e7e680738 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:57.341850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819a4fee-5bf7-4dfd-847e-2c0e7e680738 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:57.342175 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819a4fee-5bf7-4dfd-847e-2c0e7e680738 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:57.342483 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819a4fee-5bf7-4dfd-847e-2c0e7e680738 None admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:57.343338 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819a4fee-5bf7-4dfd-847e-2c0e7e680738 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:57.343655 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819a4fee-5bf7-4dfd-847e-2c0e7e680738 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:57.345801 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819a4fee-5bf7-4dfd-847e-2c0e7e680738 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:57.385181 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 251/500] 10.222.0.16 () {56 vars in 1133 bytes} [Tue Jul 12 17:19:57 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:57.390631 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819a4fee-5bf7-4dfd-847e-2c0e7e680738 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:57.416572 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f13dfb5-85b4-4ee3-b61b-a0b364d7460c None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:57.452798 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af758ed-1f79-46c6-83ff-11b14f90bdf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:57.453603 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af758ed-1f79-46c6-83ff-11b14f90bdf7 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:57.453958 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af758ed-1f79-46c6-83ff-11b14f90bdf7 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:57.454602 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af758ed-1f79-46c6-83ff-11b14f90bdf7 None admin] PATH_INFO: `/v3/users/glance` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:57.462239 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af758ed-1f79-46c6-83ff-11b14f90bdf7 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:57.463736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af758ed-1f79-46c6-83ff-11b14f90bdf7 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:57.479139 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-5af758ed-1f79-46c6-83ff-11b14f90bdf7 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Jul 12 17:19:57.479139 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:57.479139 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:57.479139 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:57.479139 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:57.479139 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:57.479139 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:57.479139 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:57.479139 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:57.479139 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:57.479139 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:57.479139 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:57.479139 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:19:57.479139 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:19:57.479139 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:19:57.479139 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:19:57.480319 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:57.480319 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:57.480319 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:19:57.480319 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:19:57.480319 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:19:57.480319 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:19:57.480319 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:57.480319 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:57.480319 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:57.480319 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:57.480319 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:57.480319 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:57.480319 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:57.480319 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:57.480319 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:57.480319 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:57.480319 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:57.480319 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:57.481301 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:57.481301 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:57.481301 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:19:57.481301 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:19:57.481301 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:19:57.481301 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:19:57.481301 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:19:57.481301 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:19:57.481301 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Jul 12 17:19:57.481301 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:57.481301 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 250/501] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:57 2022] GET /identity/v3/users/glance => generated 84 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:57.481881 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af758ed-1f79-46c6-83ff-11b14f90bdf7 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:57.501312 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819a4fee-5bf7-4dfd-847e-2c0e7e680738 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:57.532500 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:57.535266 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac554667-c8da-43fa-8a80-058f451c1552 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:57.536064 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac554667-c8da-43fa-8a80-058f451c1552 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:57.536358 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac554667-c8da-43fa-8a80-058f451c1552 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:57.536638 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac554667-c8da-43fa-8a80-058f451c1552 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:57.546855 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac554667-c8da-43fa-8a80-058f451c1552 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:57.548895 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac554667-c8da-43fa-8a80-058f451c1552 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:57.567218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-ac554667-c8da-43fa-8a80-058f451c1552 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:19:57.567218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:57.567218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:57.567218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:57.567218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:57.567218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:57.567218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:57.567218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:57.567218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:57.567218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:57.567218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:57.567218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:57.567218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:19:57.567218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:19:57.567218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:19:57.568195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:19:57.568195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:57.568195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:57.568195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:57.568195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:57.568195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:57.568195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:57.568195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:57.568195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:57.568195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:57.568195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:57.568195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:57.568195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:57.568195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:57.568195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:57.568195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:57.568195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:57.569038 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:19:57.569038 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:19:57.569038 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:19:57.569038 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:19:57.569038 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 252/502] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:19:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:57.569354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac554667-c8da-43fa-8a80-058f451c1552 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:57.581152 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:57.587464 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af758ed-1f79-46c6-83ff-11b14f90bdf7 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:57.622965 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80face7-b51e-4657-9a7f-92342879b2c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:57.623772 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80face7-b51e-4657-9a7f-92342879b2c6 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:57.624061 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80face7-b51e-4657-9a7f-92342879b2c6 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:57.624345 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80face7-b51e-4657-9a7f-92342879b2c6 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:57.635268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b80face7-b51e-4657-9a7f-92342879b2c6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:19:57.643339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-b80face7-b51e-4657-9a7f-92342879b2c6 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:19:57.643339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:57.643339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:57.643339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:57.643339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:57.643339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:57.643339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:57.643339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:57.643339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:57.643339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:57.643339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:57.643339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:57.643339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:19:57.643339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:19:57.643339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:19:57.643339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:19:57.644315 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:19:57.644315 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:19:57.644315 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:19:57.644315 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:19:57.644315 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:57.644315 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:57.644315 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:57.644315 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:57.644315 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:57.644315 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:57.644315 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:57.644315 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:57.644315 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:57.644315 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:57.644315 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:57.644315 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:57.645120 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:57.645120 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:57.645120 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:57.645120 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:57.645120 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:19:57.645120 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:19:57.645120 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:19:57.645120 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:19:57.645120 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:19:57.645120 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:19:57.645120 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:19:57.645120 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:57.645120 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 251/503] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Jul 12 17:19:57 2022] GET /identity/v3/roles/service => generated 85 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:57.646108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915cb22f-5bb6-4aa0-9c42-35ef2f5e9f7a None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:57.646108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915cb22f-5bb6-4aa0-9c42-35ef2f5e9f7a None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:57.646381 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915cb22f-5bb6-4aa0-9c42-35ef2f5e9f7a None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:57.647828 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 252/504] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:19:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:19:57.650307 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915cb22f-5bb6-4aa0-9c42-35ef2f5e9f7a None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:57.673436 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac554667-c8da-43fa-8a80-058f451c1552 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:57.707468 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d71aa18-86a1-4be1-ab93-273f9c421b8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:57.708112 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d71aa18-86a1-4be1-ab93-273f9c421b8f None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:57.708398 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d71aa18-86a1-4be1-ab93-273f9c421b8f None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:57.708677 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d71aa18-86a1-4be1-ab93-273f9c421b8f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:57.756240 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915cb22f-5bb6-4aa0-9c42-35ef2f5e9f7a None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:57.766554 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:57.776737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d71aa18-86a1-4be1-ab93-273f9c421b8f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:57.784874 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d71aa18-86a1-4be1-ab93-273f9c421b8f None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:57.788300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:57.789911 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e738e7a-b701-4b29-b92f-07c17b8b855c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:57.790611 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e738e7a-b701-4b29-b92f-07c17b8b855c None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:57.790904 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e738e7a-b701-4b29-b92f-07c17b8b855c None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:57.791188 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e738e7a-b701-4b29-b92f-07c17b8b855c None admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:57.792032 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e738e7a-b701-4b29-b92f-07c17b8b855c None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:57.792350 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e738e7a-b701-4b29-b92f-07c17b8b855c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:57.793792 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e738e7a-b701-4b29-b92f-07c17b8b855c None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:57.831536 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 253/505] 10.222.0.16 () {56 vars in 1137 bytes} [Tue Jul 12 17:19:57 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:57.832514 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e738e7a-b701-4b29-b92f-07c17b8b855c None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:57.881926 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 253/506] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:19:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 12 17:19:57.882912 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d71aa18-86a1-4be1-ab93-273f9c421b8f None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:57.935463 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e738e7a-b701-4b29-b92f-07c17b8b855c None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:57.971013 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbb4fa9-1750-43c2-9d5b-ca78561257cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:57.971680 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbb4fa9-1750-43c2-9d5b-ca78561257cd None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:57.971973 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbb4fa9-1750-43c2-9d5b-ca78561257cd None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:57.972261 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbb4fa9-1750-43c2-9d5b-ca78561257cd None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:57.972995 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbb4fa9-1750-43c2-9d5b-ca78561257cd None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:57.973314 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbb4fa9-1750-43c2-9d5b-ca78561257cd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:57.974729 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fbb4fa9-1750-43c2-9d5b-ca78561257cd None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:57.982953 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d71aa18-86a1-4be1-ab93-273f9c421b8f None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:57.987038 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 254/507] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:19:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:57.988331 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b6c48f-001e-4363-91f0-1c3afa7f93e8 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:57.988635 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b6c48f-001e-4363-91f0-1c3afa7f93e8 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:57.988923 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b6c48f-001e-4363-91f0-1c3afa7f93e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:57.993973 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:58.005405 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-54b6c48f-001e-4363-91f0-1c3afa7f93e8 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:19:58.008199 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-54b6c48f-001e-4363-91f0-1c3afa7f93e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:19:58.018717 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d448bd79-6116-4ed0-9e0f-aced7dd8514a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:58.019433 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d448bd79-6116-4ed0-9e0f-aced7dd8514a None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:58.019716 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d448bd79-6116-4ed0-9e0f-aced7dd8514a None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:58.019995 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d448bd79-6116-4ed0-9e0f-aced7dd8514a None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:58.020750 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d448bd79-6116-4ed0-9e0f-aced7dd8514a None admin] RBAC: Adding query filter params (name=service) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:58.021063 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d448bd79-6116-4ed0-9e0f-aced7dd8514a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:58.022448 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d448bd79-6116-4ed0-9e0f-aced7dd8514a None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:58.029550 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:58.033139 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 254/508] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Jul 12 17:19:57 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:58.033999 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d448bd79-6116-4ed0-9e0f-aced7dd8514a None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:58.077490 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 255/509] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:19:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:19:58.078313 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b6c48f-001e-4363-91f0-1c3afa7f93e8 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:58.137047 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d448bd79-6116-4ed0-9e0f-aced7dd8514a None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:58.172745 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa09bc56-aefa-440e-a5c1-048105095c6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:58.173397 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa09bc56-aefa-440e-a5c1-048105095c6a None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:58.173684 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa09bc56-aefa-440e-a5c1-048105095c6a None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:58.173971 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa09bc56-aefa-440e-a5c1-048105095c6a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:58.181376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b6c48f-001e-4363-91f0-1c3afa7f93e8 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:58.214097 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:58.217108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51b4514-03c9-4b9a-96dc-042e49fa17a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:58.217902 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51b4514-03c9-4b9a-96dc-042e49fa17a2 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:58.218191 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51b4514-03c9-4b9a-96dc-042e49fa17a2 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:58.218501 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51b4514-03c9-4b9a-96dc-042e49fa17a2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:58.229912 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a51b4514-03c9-4b9a-96dc-042e49fa17a2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:58.231485 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a51b4514-03c9-4b9a-96dc-042e49fa17a2 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:58.244154 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa09bc56-aefa-440e-a5c1-048105095c6a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:58.245614 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa09bc56-aefa-440e-a5c1-048105095c6a None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:58.248782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-a51b4514-03c9-4b9a-96dc-042e49fa17a2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:19:58.248782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:58.248782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:58.248782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:58.248782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:58.248782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:58.248782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:58.248782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:58.248782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:58.248782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:58.248782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:58.248782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:58.248782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:19:58.248782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:19:58.248782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:19:58.249813 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:19:58.249813 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:58.249813 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:58.249813 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:58.249813 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:58.249813 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:58.249813 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:58.249813 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:58.249813 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:58.249813 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:58.249813 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:58.249813 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:58.249813 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:58.249813 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:58.249813 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:58.249813 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:58.249813 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:58.250903 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:19:58.250903 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:19:58.250903 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:19:58.250903 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:58.250903 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 256/510] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:19:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:58.251590 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51b4514-03c9-4b9a-96dc-042e49fa17a2 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:58.265909 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:58.345862 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 255/511] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:19:58 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 12 17:19:58.347215 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa09bc56-aefa-440e-a5c1-048105095c6a None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:58.356476 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51b4514-03c9-4b9a-96dc-042e49fa17a2 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:58.392819 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b398c6c9-c502-40f5-8b1b-f58137fd3643 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:58.394390 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b398c6c9-c502-40f5-8b1b-f58137fd3643 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:58.394907 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b398c6c9-c502-40f5-8b1b-f58137fd3643 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:58.395375 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b398c6c9-c502-40f5-8b1b-f58137fd3643 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:58.406692 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b398c6c9-c502-40f5-8b1b-f58137fd3643 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:58.409702 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b398c6c9-c502-40f5-8b1b-f58137fd3643 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:58.434694 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-b398c6c9-c502-40f5-8b1b-f58137fd3643 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jul 12 17:19:58.434694 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:58.434694 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:58.434694 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:58.434694 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:58.434694 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:58.434694 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:58.434694 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:58.434694 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:58.434694 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:58.434694 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:58.434694 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:58.434694 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:19:58.434694 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:19:58.434694 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:19:58.434694 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:19:58.435834 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:58.435834 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:58.435834 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:19:58.435834 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:19:58.435834 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:19:58.435834 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:19:58.435834 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:58.435834 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:58.435834 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:58.435834 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:58.435834 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:58.435834 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:58.435834 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:58.435834 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:58.435834 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:58.435834 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:58.435834 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:58.435834 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:58.436938 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:58.436938 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:58.436938 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:19:58.436938 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:19:58.436938 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:19:58.436938 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:19:58.436938 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:19:58.436938 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:19:58.436938 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jul 12 17:19:58.436938 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:58.436938 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 257/512] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:19:58 2022] GET /identity/v3/users/cinder => generated 84 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:58.438596 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b398c6c9-c502-40f5-8b1b-f58137fd3643 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:58.442997 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:58.474724 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa09bc56-aefa-440e-a5c1-048105095c6a None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:58.509526 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07564d14-218c-427d-9e05-d9723ae7b7a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:58.510086 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:58.510313 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07564d14-218c-427d-9e05-d9723ae7b7a4 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:58.510626 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07564d14-218c-427d-9e05-d9723ae7b7a4 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:58.510912 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07564d14-218c-427d-9e05-d9723ae7b7a4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:58.522088 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07564d14-218c-427d-9e05-d9723ae7b7a4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:58.523637 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07564d14-218c-427d-9e05-d9723ae7b7a4 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:58.540978 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-07564d14-218c-427d-9e05-d9723ae7b7a4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:19:58.540978 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:58.540978 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:58.540978 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:58.540978 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:58.540978 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:58.540978 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:58.540978 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:58.540978 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:58.540978 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:58.540978 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:58.540978 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:58.540978 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:19:58.540978 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:19:58.540978 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:19:58.542013 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:19:58.542013 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:58.542013 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:58.542013 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:58.542013 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:58.542013 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:58.542013 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:58.542013 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:58.542013 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:58.542013 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:58.542013 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:58.542013 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:58.542013 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:58.542013 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:58.542013 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:58.542013 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:58.542013 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:58.547695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:19:58.547695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:19:58.547695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:19:58.547695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:19:58.547905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b398c6c9-c502-40f5-8b1b-f58137fd3643 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:58.547987 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 256/513] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:19:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:58.547987 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07564d14-218c-427d-9e05-d9723ae7b7a4 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:58.578107 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26cb2b5a-f1ee-4e45-acf3-d21f8d4ec77a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:58.579537 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cb2b5a-f1ee-4e45-acf3-d21f8d4ec77a None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:58.579826 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cb2b5a-f1ee-4e45-acf3-d21f8d4ec77a None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:58.580094 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cb2b5a-f1ee-4e45-acf3-d21f8d4ec77a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:58.590781 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26cb2b5a-f1ee-4e45-acf3-d21f8d4ec77a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:58.592282 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26cb2b5a-f1ee-4e45-acf3-d21f8d4ec77a None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:58.610584 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-26cb2b5a-f1ee-4e45-acf3-d21f8d4ec77a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:19:58.610584 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:58.610584 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:58.610584 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:58.610584 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:58.610584 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:58.610584 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:58.610584 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:58.610584 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:58.610584 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:58.610584 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:58.610584 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:58.610584 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:19:58.610584 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:19:58.610584 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:19:58.611559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:19:58.611559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:58.611559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:58.611559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:58.611559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:58.611559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:58.611559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:58.611559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:58.611559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:58.611559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:58.611559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:58.611559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:58.611559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:58.611559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:58.611559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:58.611559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:58.611559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:58.612439 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:19:58.612439 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:19:58.612439 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:19:58.612439 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:58.612439 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 258/514] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:19:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:58.612700 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26cb2b5a-f1ee-4e45-acf3-d21f8d4ec77a None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:58.652645 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07564d14-218c-427d-9e05-d9723ae7b7a4 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:58.676405 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:58.687687 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd95a68a-65ce-4704-936b-17c94a5efd17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:58.688353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd95a68a-65ce-4704-936b-17c94a5efd17 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:58.688646 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd95a68a-65ce-4704-936b-17c94a5efd17 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:58.688928 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd95a68a-65ce-4704-936b-17c94a5efd17 None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:58.689662 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd95a68a-65ce-4704-936b-17c94a5efd17 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:58.690181 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd95a68a-65ce-4704-936b-17c94a5efd17 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:58.691606 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd95a68a-65ce-4704-936b-17c94a5efd17 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:58.703502 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 257/515] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:19:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:58.704577 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd95a68a-65ce-4704-936b-17c94a5efd17 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:58.715292 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26cb2b5a-f1ee-4e45-acf3-d21f8d4ec77a None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:58.741299 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:58.752279 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4a2144-4dc5-4f55-9ec3-9d60afa2c70a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:58.753075 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4a2144-4dc5-4f55-9ec3-9d60afa2c70a None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:58.753452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4a2144-4dc5-4f55-9ec3-9d60afa2c70a None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:58.753723 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4a2144-4dc5-4f55-9ec3-9d60afa2c70a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:58.763760 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e4a2144-4dc5-4f55-9ec3-9d60afa2c70a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:58.765250 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e4a2144-4dc5-4f55-9ec3-9d60afa2c70a None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:58.783861 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-4e4a2144-4dc5-4f55-9ec3-9d60afa2c70a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:19:58.783861 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:58.783861 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:58.783861 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:58.783861 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:58.783861 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:58.783861 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:58.783861 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:58.783861 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:58.783861 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:58.783861 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:58.783861 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:58.783861 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:19:58.783861 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:19:58.783861 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:19:58.784921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:19:58.784921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:58.784921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:58.784921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:58.784921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:58.784921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:58.784921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:58.784921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:58.784921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:58.784921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:58.784921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:58.784921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:58.784921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:58.784921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:58.784921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:58.784921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:58.784921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:58.790389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:19:58.790389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:19:58.790389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:19:58.790389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:58.790389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 259/516] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:19:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:58.790389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4a2144-4dc5-4f55-9ec3-9d60afa2c70a None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:58.816602 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd95a68a-65ce-4704-936b-17c94a5efd17 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:58.851104 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8302ed0-ef95-4b7f-8676-85d22b539b3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:58.851770 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8302ed0-ef95-4b7f-8676-85d22b539b3d None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:58.852060 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8302ed0-ef95-4b7f-8676-85d22b539b3d None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:58.852343 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8302ed0-ef95-4b7f-8676-85d22b539b3d None admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:58.853277 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8302ed0-ef95-4b7f-8676-85d22b539b3d None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:58.853594 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8302ed0-ef95-4b7f-8676-85d22b539b3d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:58.855305 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8302ed0-ef95-4b7f-8676-85d22b539b3d None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:58.872420 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 258/517] 10.222.0.16 () {56 vars in 1137 bytes} [Tue Jul 12 17:19:58 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:58.873247 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8302ed0-ef95-4b7f-8676-85d22b539b3d None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:58.894191 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4a2144-4dc5-4f55-9ec3-9d60afa2c70a None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:58.915287 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:58.929358 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11cd95c3-c3ed-4729-98e7-7dd7ef68b484 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:58.930027 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cd95c3-c3ed-4729-98e7-7dd7ef68b484 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:58.930317 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cd95c3-c3ed-4729-98e7-7dd7ef68b484 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:58.930639 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cd95c3-c3ed-4729-98e7-7dd7ef68b484 None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:58.931368 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11cd95c3-c3ed-4729-98e7-7dd7ef68b484 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:58.931686 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11cd95c3-c3ed-4729-98e7-7dd7ef68b484 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:58.933146 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11cd95c3-c3ed-4729-98e7-7dd7ef68b484 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:58.944461 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 260/518] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:19:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:58.945309 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11cd95c3-c3ed-4729-98e7-7dd7ef68b484 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:58.973890 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8302ed0-ef95-4b7f-8676-85d22b539b3d None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:58.981531 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:59.010959 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f55fe445-40e4-4135-a0b5-d29394a2944c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:59.011621 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55fe445-40e4-4135-a0b5-d29394a2944c None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:59.011908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55fe445-40e4-4135-a0b5-d29394a2944c None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:59.012188 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55fe445-40e4-4135-a0b5-d29394a2944c None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:59.012941 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f55fe445-40e4-4135-a0b5-d29394a2944c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:59.013256 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f55fe445-40e4-4135-a0b5-d29394a2944c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:59.014642 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f55fe445-40e4-4135-a0b5-d29394a2944c None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:59.026537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 259/519] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:19:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:59.027342 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f55fe445-40e4-4135-a0b5-d29394a2944c None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:59.054543 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11cd95c3-c3ed-4729-98e7-7dd7ef68b484 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:59.088266 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976f22ca-8b2c-4c1d-b380-5634c0b12297 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:59.089054 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976f22ca-8b2c-4c1d-b380-5634c0b12297 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:59.089338 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976f22ca-8b2c-4c1d-b380-5634c0b12297 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:59.089620 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976f22ca-8b2c-4c1d-b380-5634c0b12297 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:59.099099 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-976f22ca-8b2c-4c1d-b380-5634c0b12297 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:59.100571 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-976f22ca-8b2c-4c1d-b380-5634c0b12297 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:59.115654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-976f22ca-8b2c-4c1d-b380-5634c0b12297 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:19:59.115654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:59.115654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:59.115654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:59.115654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:59.115654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:59.115654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:59.115654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:59.115654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:59.115654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:59.115654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:59.115654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:59.115654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:19:59.115654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:19:59.115654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:19:59.116642 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:19:59.116642 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:59.116642 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:59.116642 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:59.116642 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:59.116642 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:59.116642 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:59.116642 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:59.116642 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:59.116642 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:59.116642 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:59.116642 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:59.116642 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:59.116642 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:59.116642 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:59.116642 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:59.116642 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:59.118053 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:19:59.118053 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:19:59.118053 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:19:59.118053 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:19:59.118053 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:19:59.118053 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:19:59.118053 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:19:59.118053 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:59.118053 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 261/520] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:19:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:59.118053 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976f22ca-8b2c-4c1d-b380-5634c0b12297 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:59.129648 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:59.133138 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f55fe445-40e4-4135-a0b5-d29394a2944c None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:59.167460 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139c7593-0bfe-4d56-b555-4843e1d6d848 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:59.168130 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139c7593-0bfe-4d56-b555-4843e1d6d848 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:59.168415 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139c7593-0bfe-4d56-b555-4843e1d6d848 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:59.168698 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139c7593-0bfe-4d56-b555-4843e1d6d848 None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:59.169417 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139c7593-0bfe-4d56-b555-4843e1d6d848 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:59.169726 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139c7593-0bfe-4d56-b555-4843e1d6d848 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:59.171129 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139c7593-0bfe-4d56-b555-4843e1d6d848 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:59.183277 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 260/521] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:19:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:59.184251 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139c7593-0bfe-4d56-b555-4843e1d6d848 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:59.213552 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:59.220392 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976f22ca-8b2c-4c1d-b380-5634c0b12297 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:59.255118 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26dd5460-b7ae-4819-8485-c94efd2698d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:59.255782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dd5460-b7ae-4819-8485-c94efd2698d0 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:59.256071 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dd5460-b7ae-4819-8485-c94efd2698d0 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:59.256354 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26dd5460-b7ae-4819-8485-c94efd2698d0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:59.287523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139c7593-0bfe-4d56-b555-4843e1d6d848 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:59.322173 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c2d0b4-17a7-4353-a3ea-c1446b40199e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:59.322937 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c2d0b4-17a7-4353-a3ea-c1446b40199e None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:59.323223 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c2d0b4-17a7-4353-a3ea-c1446b40199e None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:59.323514 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c2d0b4-17a7-4353-a3ea-c1446b40199e None admin] PATH_INFO: `/v3/users/neutron` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:59.327253 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26dd5460-b7ae-4819-8485-c94efd2698d0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:59.330158 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26dd5460-b7ae-4819-8485-c94efd2698d0 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:59.331390 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c2d0b4-17a7-4353-a3ea-c1446b40199e None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:59.332834 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c2d0b4-17a7-4353-a3ea-c1446b40199e None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:59.348684 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-e2c2d0b4-17a7-4353-a3ea-c1446b40199e None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Jul 12 17:19:59.348684 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:59.348684 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:59.348684 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:59.348684 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:59.348684 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:59.348684 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:59.348684 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:59.348684 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:59.348684 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:59.348684 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:59.348684 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:59.348684 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:19:59.348684 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:19:59.348684 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:19:59.348684 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:19:59.349777 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:59.349777 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:59.349777 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:19:59.349777 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:19:59.349777 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:19:59.349777 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:19:59.349777 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:59.349777 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:59.349777 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:59.349777 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:59.349777 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:59.349777 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:59.349777 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:59.349777 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:59.349777 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:59.349777 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:59.349777 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:59.349777 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:59.351129 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:59.351129 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:59.351129 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:19:59.351129 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:19:59.351129 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:19:59.351129 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:19:59.351129 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:19:59.351129 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:19:59.351129 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Jul 12 17:19:59.351129 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:19:59.351129 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 261/522] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Jul 12 17:19:59 2022] GET /identity/v3/users/neutron => generated 85 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:59.351129 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c2d0b4-17a7-4353-a3ea-c1446b40199e None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:59.376512 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:59.424583 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:59.431826 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 262/523] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:19:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 12 17:19:59.432638 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26dd5460-b7ae-4819-8485-c94efd2698d0 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:59.456558 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c2d0b4-17a7-4353-a3ea-c1446b40199e None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:59.491097 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d68a8ec-aacd-4420-8e4d-b782ff809be1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:59.491772 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d68a8ec-aacd-4420-8e4d-b782ff809be1 None admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:59.492061 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d68a8ec-aacd-4420-8e4d-b782ff809be1 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:59.492349 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d68a8ec-aacd-4420-8e4d-b782ff809be1 None admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:59.493056 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d68a8ec-aacd-4420-8e4d-b782ff809be1 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:59.494646 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d68a8ec-aacd-4420-8e4d-b782ff809be1 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:59.532940 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26dd5460-b7ae-4819-8485-c94efd2698d0 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:59.566075 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e103fb-86b3-4710-80cd-12425f3d0190 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:59.566769 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e103fb-86b3-4710-80cd-12425f3d0190 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:59.567060 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e103fb-86b3-4710-80cd-12425f3d0190 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:59.567344 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e103fb-86b3-4710-80cd-12425f3d0190 None admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:59.568246 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e103fb-86b3-4710-80cd-12425f3d0190 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:59.568566 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e103fb-86b3-4710-80cd-12425f3d0190 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:59.569990 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e103fb-86b3-4710-80cd-12425f3d0190 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:59.588624 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 263/524] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Jul 12 17:19:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:59.589439 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e103fb-86b3-4710-80cd-12425f3d0190 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:59.620921 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 262/525] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Jul 12 17:19:59 2022] POST /identity/v3/users => generated 247 bytes in 270 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:19:59.622042 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d68a8ec-aacd-4420-8e4d-b782ff809be1 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:59.652533 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:59.706067 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e103fb-86b3-4710-80cd-12425f3d0190 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:59.712966 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:59.739557 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d68a8ec-aacd-4420-8e4d-b782ff809be1 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:59.746901 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64f1171-84b5-454b-88d5-bf594926142e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:59.747592 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64f1171-84b5-454b-88d5-bf594926142e None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:59.747883 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64f1171-84b5-454b-88d5-bf594926142e None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:59.748168 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64f1171-84b5-454b-88d5-bf594926142e None admin] PATH_INFO: `/v3/projects/service` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:59.757113 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64f1171-84b5-454b-88d5-bf594926142e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:59.761310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64f1171-84b5-454b-88d5-bf594926142e None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:59.774292 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-e64f1171-84b5-454b-88d5-bf594926142e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:19:59.774292 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:59.774292 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:59.774292 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:59.774292 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:59.774292 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:59.774292 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:59.774292 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:59.774292 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:59.774292 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:59.774292 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:59.774292 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:59.774292 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:19:59.774292 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:19:59.774292 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:19:59.775641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:19:59.775641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:59.775641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:59.775641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:59.775641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:59.775641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:59.775641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:59.775641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:59.775641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:59.775641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:59.775641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:59.775641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:59.775641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:59.775641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:59.775641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:59.775641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:59.775641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:59.776486 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:19:59.776486 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:19:59.776486 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:19:59.776486 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:19:59.776486 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:19:59.776486 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:19:59.776486 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:19:59.776486 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:59.778721 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f95358f-221b-4b52-81ca-8c20304ae7d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:59.781283 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f95358f-221b-4b52-81ca-8c20304ae7d2 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:59.781405 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 264/526] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:19:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:59.781481 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f95358f-221b-4b52-81ca-8c20304ae7d2 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:59.781481 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f95358f-221b-4b52-81ca-8c20304ae7d2 None admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:59.781635 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64f1171-84b5-454b-88d5-bf594926142e None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:59.781749 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f95358f-221b-4b52-81ca-8c20304ae7d2 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:59.781984 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f95358f-221b-4b52-81ca-8c20304ae7d2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:59.783704 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f95358f-221b-4b52-81ca-8c20304ae7d2 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:59.804559 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 263/527] 10.222.0.16 () {56 vars in 1139 bytes} [Tue Jul 12 17:19:59 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:59.805405 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f95358f-221b-4b52-81ca-8c20304ae7d2 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:59.901026 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64f1171-84b5-454b-88d5-bf594926142e None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:59.915921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:59.918194 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f95358f-221b-4b52-81ca-8c20304ae7d2 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:19:59.937453 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0de2691-a496-44b7-bafe-cae1487c2f54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:59.939425 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0de2691-a496-44b7-bafe-cae1487c2f54 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:59.939904 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0de2691-a496-44b7-bafe-cae1487c2f54 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:59.940322 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0de2691-a496-44b7-bafe-cae1487c2f54 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:59.950883 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0de2691-a496-44b7-bafe-cae1487c2f54 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:59.952772 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0de2691-a496-44b7-bafe-cae1487c2f54 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:59.955228 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e96ad2-7746-419d-88f5-aef921aa6c89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:19:59.955911 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e96ad2-7746-419d-88f5-aef921aa6c89 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:19:59.956500 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e96ad2-7746-419d-88f5-aef921aa6c89 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:19:59.956785 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e96ad2-7746-419d-88f5-aef921aa6c89 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:19:59.957635 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67e96ad2-7746-419d-88f5-aef921aa6c89 None admin] RBAC: Adding query filter params (role.id=bef74d4975164f75bcec96ac47afea73, scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2, user.id=9012d5b88f554dc596582a7693c2a72a) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:19:59.957952 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67e96ad2-7746-419d-88f5-aef921aa6c89 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:19:59.959431 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67e96ad2-7746-419d-88f5-aef921aa6c89 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:19:59.961826 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:19:59.966503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 264/528] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:19:59 2022] GET /identity/v3/role_assignments?user.id=9012d5b88f554dc596582a7693c2a72a&scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2&role.id=bef74d4975164f75bcec96ac47afea73 => generated 259 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:19:59.967327 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e96ad2-7746-419d-88f5-aef921aa6c89 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:19:59.973637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-a0de2691-a496-44b7-bafe-cae1487c2f54 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:19:59.973637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:19:59.973637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:19:59.973637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:19:59.973637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:19:59.973637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:19:59.973637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:19:59.973637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:19:59.973637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:19:59.973637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:19:59.973637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:19:59.973637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:19:59.973637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:19:59.973637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:19:59.973637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:19:59.974962 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:19:59.974962 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:19:59.974962 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:19:59.974962 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:19:59.974962 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:19:59.974962 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:19:59.974962 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:19:59.974962 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:19:59.974962 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:19:59.974962 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:19:59.974962 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:19:59.974962 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:19:59.974962 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:19:59.974962 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:19:59.974962 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:19:59.974962 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:19:59.974962 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:19:59.980978 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:19:59.980978 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:19:59.980978 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:19:59.980978 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:19:59.980978 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 265/529] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:19:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:19:59.980978 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0de2691-a496-44b7-bafe-cae1487c2f54 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:00.105059 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e96ad2-7746-419d-88f5-aef921aa6c89 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:00.123275 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0de2691-a496-44b7-bafe-cae1487c2f54 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:00.147784 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5068df6-c1d9-4c71-b8ca-f2ccb4914506 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:00.148450 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5068df6-c1d9-4c71-b8ca-f2ccb4914506 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:00.148737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5068df6-c1d9-4c71-b8ca-f2ccb4914506 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:00.149022 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5068df6-c1d9-4c71-b8ca-f2ccb4914506 None admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:00.149818 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5068df6-c1d9-4c71-b8ca-f2ccb4914506 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:00.150132 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5068df6-c1d9-4c71-b8ca-f2ccb4914506 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:00.151657 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5068df6-c1d9-4c71-b8ca-f2ccb4914506 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:00.164930 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67074bb-fa7a-47d0-b353-3dad8b3d0b5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:00.165914 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67074bb-fa7a-47d0-b353-3dad8b3d0b5f None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:00.166488 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67074bb-fa7a-47d0-b353-3dad8b3d0b5f None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:00.166958 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67074bb-fa7a-47d0-b353-3dad8b3d0b5f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:00.169832 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 265/530] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Jul 12 17:19:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:00.174837 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5068df6-c1d9-4c71-b8ca-f2ccb4914506 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:00.210216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:00.236618 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:00.251970 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b67074bb-fa7a-47d0-b353-3dad8b3d0b5f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:00.254291 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b67074bb-fa7a-47d0-b353-3dad8b3d0b5f None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:00.280492 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5068df6-c1d9-4c71-b8ca-f2ccb4914506 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:00.322374 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93497790-d16b-4259-b51d-9d389de544fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:00.323077 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93497790-d16b-4259-b51d-9d389de544fd None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:00.323394 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93497790-d16b-4259-b51d-9d389de544fd None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:00.323926 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93497790-d16b-4259-b51d-9d389de544fd None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:00.324543 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93497790-d16b-4259-b51d-9d389de544fd None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:00.324861 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93497790-d16b-4259-b51d-9d389de544fd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:00.326254 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93497790-d16b-4259-b51d-9d389de544fd None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:00.340637 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 266/531] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:00.341532 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93497790-d16b-4259-b51d-9d389de544fd None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:00.397692 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 266/532] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:19:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 420 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 12 17:20:00.398939 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67074bb-fa7a-47d0-b353-3dad8b3d0b5f None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:00.461576 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93497790-d16b-4259-b51d-9d389de544fd None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:00.481776 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:00.492108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:00.504559 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01cf3c5f-a8bf-40df-bead-bafb6e7dcf3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:00.505460 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cf3c5f-a8bf-40df-bead-bafb6e7dcf3e None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:00.505964 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cf3c5f-a8bf-40df-bead-bafb6e7dcf3e None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:00.506490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cf3c5f-a8bf-40df-bead-bafb6e7dcf3e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:00.507556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01cf3c5f-a8bf-40df-bead-bafb6e7dcf3e None admin] RBAC: Adding query filter params (role.id=bef74d4975164f75bcec96ac47afea73, scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2, user.id=4018d94a1feb4d6dade88ee9663f9729) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:00.508075 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01cf3c5f-a8bf-40df-bead-bafb6e7dcf3e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:00.509778 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01cf3c5f-a8bf-40df-bead-bafb6e7dcf3e None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:00.516120 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 267/533] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:20:00 2022] GET /identity/v3/role_assignments?user.id=4018d94a1feb4d6dade88ee9663f9729&scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2&role.id=bef74d4975164f75bcec96ac47afea73 => generated 259 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:00.517680 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01cf3c5f-a8bf-40df-bead-bafb6e7dcf3e None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:00.557816 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67074bb-fa7a-47d0-b353-3dad8b3d0b5f None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:00.623138 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde328b8-9401-48bc-966a-aaa203183539 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:00.623945 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde328b8-9401-48bc-966a-aaa203183539 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:00.624234 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde328b8-9401-48bc-966a-aaa203183539 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:00.624517 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde328b8-9401-48bc-966a-aaa203183539 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:00.633926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fde328b8-9401-48bc-966a-aaa203183539 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:00.635625 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fde328b8-9401-48bc-966a-aaa203183539 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:00.655214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-fde328b8-9401-48bc-966a-aaa203183539 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:00.655214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:00.655214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:00.655214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:00.655214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:00.655214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:00.655214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:00.655214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:00.655214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:00.655214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:00.655214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:00.655214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:00.655214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:20:00.655214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:20:00.655214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:20:00.656209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:20:00.656209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:00.656209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:00.656209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:00.656209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:00.656209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:00.656209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:00.656209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:00.656209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:00.656209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:00.656209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:00.656209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:00.656209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:00.656209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:00.656209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:00.656209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:00.656209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:00.657152 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:20:00.657152 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:20:00.657152 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:20:00.657152 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:20:00.657152 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:20:00.657152 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:20:00.657152 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:00.657152 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:00.657152 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 267/534] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:20:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 258 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:00.662511 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde328b8-9401-48bc-966a-aaa203183539 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:00.775679 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01cf3c5f-a8bf-40df-bead-bafb6e7dcf3e None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:00.789497 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde328b8-9401-48bc-966a-aaa203183539 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:00.820002 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61904b6e-9ba0-415a-abe0-3e97baaf37dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:00.821225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61904b6e-9ba0-415a-abe0-3e97baaf37dd None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:00.821751 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61904b6e-9ba0-415a-abe0-3e97baaf37dd None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:00.822252 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61904b6e-9ba0-415a-abe0-3e97baaf37dd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:00.824788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:00.836272 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61904b6e-9ba0-415a-abe0-3e97baaf37dd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:00.838648 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d422877c-b060-436a-8a02-a3fb8d6ec6b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:00.838774 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61904b6e-9ba0-415a-abe0-3e97baaf37dd None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:00.838871 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d422877c-b060-436a-8a02-a3fb8d6ec6b9 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:00.838871 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d422877c-b060-436a-8a02-a3fb8d6ec6b9 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:00.839036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d422877c-b060-436a-8a02-a3fb8d6ec6b9 None admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:00.839757 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d422877c-b060-436a-8a02-a3fb8d6ec6b9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:00.840078 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d422877c-b060-436a-8a02-a3fb8d6ec6b9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:00.841790 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d422877c-b060-436a-8a02-a3fb8d6ec6b9 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:00.866496 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 268/535] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Jul 12 17:20:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:00.866695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-61904b6e-9ba0-415a-abe0-3e97baaf37dd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:00.866695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:00.866695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:00.866695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:00.866695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:00.866695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:00.866695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:00.866695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:00.866695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:00.866695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:00.866695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:00.866695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:00.866695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:00.866695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:00.866695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:00.867511 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:00.867511 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:00.867511 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:00.867511 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:00.867511 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:00.867511 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:00.867511 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:00.867511 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:00.867511 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:00.867511 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:00.867511 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:00.867511 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:00.867511 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:00.867511 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:00.867511 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:00.867511 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:00.867511 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:00.868353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:00.868353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:00.868353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:00.868353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:00.868353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 268/536] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 344 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:00.868653 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d422877c-b060-436a-8a02-a3fb8d6ec6b9 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:00.868728 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61904b6e-9ba0-415a-abe0-3e97baaf37dd None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:00.950848 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:01.037129 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61904b6e-9ba0-415a-abe0-3e97baaf37dd None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:01.080867 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241756d6-8412-46be-bc2e-3fbec166d469 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:01.081549 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241756d6-8412-46be-bc2e-3fbec166d469 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:01.081838 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241756d6-8412-46be-bc2e-3fbec166d469 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:01.082122 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241756d6-8412-46be-bc2e-3fbec166d469 None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:01.083845 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-241756d6-8412-46be-bc2e-3fbec166d469 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:01.084162 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-241756d6-8412-46be-bc2e-3fbec166d469 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:01.085552 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-241756d6-8412-46be-bc2e-3fbec166d469 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:01.109160 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 269/537] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:01.113177 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d422877c-b060-436a-8a02-a3fb8d6ec6b9 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:01.115557 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241756d6-8412-46be-bc2e-3fbec166d469 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:01.186969 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2846d60-c5cf-4d23-b8c6-deadca000cef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:01.187634 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2846d60-c5cf-4d23-b8c6-deadca000cef None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:01.188105 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2846d60-c5cf-4d23-b8c6-deadca000cef None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:01.188558 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2846d60-c5cf-4d23-b8c6-deadca000cef None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:01.189559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2846d60-c5cf-4d23-b8c6-deadca000cef None admin] RBAC: Adding query filter params (role.id=bef74d4975164f75bcec96ac47afea73, scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2, user.id=f6a82b49464846d1a207189f17fd4cf7) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:01.189880 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2846d60-c5cf-4d23-b8c6-deadca000cef None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:01.191338 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2846d60-c5cf-4d23-b8c6-deadca000cef None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:01.210580 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 269/538] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:20:00 2022] GET /identity/v3/role_assignments?user.id=f6a82b49464846d1a207189f17fd4cf7&scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2&role.id=bef74d4975164f75bcec96ac47afea73 => generated 259 bytes in 339 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:01.274723 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241756d6-8412-46be-bc2e-3fbec166d469 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:01.295033 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:01.312490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8207d9e5-81a8-4d5d-a442-72ab260e1eda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:01.313440 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8207d9e5-81a8-4d5d-a442-72ab260e1eda None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:01.313771 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8207d9e5-81a8-4d5d-a442-72ab260e1eda None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:01.314056 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8207d9e5-81a8-4d5d-a442-72ab260e1eda None admin] PATH_INFO: `/v3/projects/service` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:01.321727 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8207d9e5-81a8-4d5d-a442-72ab260e1eda None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:01.323201 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8207d9e5-81a8-4d5d-a442-72ab260e1eda None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:01.339999 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-8207d9e5-81a8-4d5d-a442-72ab260e1eda None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:01.339999 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:01.339999 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:01.339999 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:01.339999 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:01.339999 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:01.339999 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:01.339999 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:01.339999 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:01.339999 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:01.339999 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:01.339999 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:01.339999 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:20:01.339999 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:20:01.339999 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:20:01.340927 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:20:01.340927 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:01.340927 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:01.340927 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:01.340927 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:01.340927 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:01.340927 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:01.340927 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:01.340927 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:01.340927 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:01.340927 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:01.340927 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:01.340927 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:01.340927 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:01.340927 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:01.340927 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:01.340927 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:01.341787 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:20:01.341787 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:20:01.341787 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:20:01.341787 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:20:01.341787 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:20:01.341787 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:20:01.341787 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:01.341787 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:01.341787 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 270/539] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:20:01 2022] GET /identity/v3/projects/service => generated 88 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:01.348125 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2846d60-c5cf-4d23-b8c6-deadca000cef None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:01.407493 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:01.459405 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2846d60-c5cf-4d23-b8c6-deadca000cef None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:01.495583 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939b1636-c062-426e-810b-fed60b1b2b67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:01.496256 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939b1636-c062-426e-810b-fed60b1b2b67 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:01.496543 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939b1636-c062-426e-810b-fed60b1b2b67 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:01.496825 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939b1636-c062-426e-810b-fed60b1b2b67 None admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:01.497614 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939b1636-c062-426e-810b-fed60b1b2b67 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:01.497926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939b1636-c062-426e-810b-fed60b1b2b67 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:01.499391 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939b1636-c062-426e-810b-fed60b1b2b67 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:01.512578 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 270/540] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Jul 12 17:20:01 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:01.518375 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8207d9e5-81a8-4d5d-a442-72ab260e1eda None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:01.625852 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8207d9e5-81a8-4d5d-a442-72ab260e1eda None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:01.661001 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df73421-d4aa-45cf-aa31-ce9965aba55a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:01.661658 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df73421-d4aa-45cf-aa31-ce9965aba55a None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:01.661943 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df73421-d4aa-45cf-aa31-ce9965aba55a None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:01.662218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df73421-d4aa-45cf-aa31-ce9965aba55a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:01.663086 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df73421-d4aa-45cf-aa31-ce9965aba55a None admin] RBAC: Adding query filter params (role.id=bef74d4975164f75bcec96ac47afea73, scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2, user.id=bf5bf02ae13d4d86a1b157369ef07a00) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:01.663398 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df73421-d4aa-45cf-aa31-ce9965aba55a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:01.664825 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df73421-d4aa-45cf-aa31-ce9965aba55a None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:01.670941 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 271/541] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:20:01 2022] GET /identity/v3/role_assignments?user.id=bf5bf02ae13d4d86a1b157369ef07a00&scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2&role.id=bef74d4975164f75bcec96ac47afea73 => generated 259 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:01.837257 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7560ef8-1cc3-449f-9472-cf6567c81d10 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:01.837745 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7560ef8-1cc3-449f-9472-cf6567c81d10 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:01.837745 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7560ef8-1cc3-449f-9472-cf6567c81d10 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:01.839124 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 271/542] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:01.844961 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b71f6d9-d29e-4440-8fb1-7e8740e57135 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:01.845310 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b71f6d9-d29e-4440-8fb1-7e8740e57135 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:01.845520 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b71f6d9-d29e-4440-8fb1-7e8740e57135 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:01.867548 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-1b71f6d9-d29e-4440-8fb1-7e8740e57135 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:01.868791 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-1b71f6d9-d29e-4440-8fb1-7e8740e57135 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:01.888866 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:01.949683 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 272/543] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:20:01.997100 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7560ef8-1cc3-449f-9472-cf6567c81d10 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:02.128322 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7560ef8-1cc3-449f-9472-cf6567c81d10 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:02.140492 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:02.177981 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2732e1-196c-4086-bbb9-a2f6aa86fdf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:02.179104 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2732e1-196c-4086-bbb9-a2f6aa86fdf1 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:02.179600 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2732e1-196c-4086-bbb9-a2f6aa86fdf1 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:02.180209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2732e1-196c-4086-bbb9-a2f6aa86fdf1 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:02.195904 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2732e1-196c-4086-bbb9-a2f6aa86fdf1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:20:02.204723 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-0a2732e1-196c-4086-bbb9-a2f6aa86fdf1 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:20:02.204723 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:02.204723 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:02.204723 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:02.204723 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:02.204723 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:02.204723 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:02.204723 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:02.204723 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:02.204723 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:02.204723 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:02.204723 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:02.204723 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:20:02.204723 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:20:02.204723 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:20:02.204723 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:20:02.205609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:20:02.205609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:20:02.205609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:20:02.205609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:20:02.205609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:02.205609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:02.205609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:02.205609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:02.205609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:02.205609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:02.205609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:02.205609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:02.205609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:02.205609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:02.205609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:02.205609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:02.206841 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:02.206841 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:02.206841 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:02.206841 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:02.206841 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:20:02.206841 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:20:02.206841 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:20:02.206841 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:20:02.206841 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:20:02.206841 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:20:02.206841 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:20:02.206841 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:02.206841 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 272/544] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Jul 12 17:20:01 2022] GET /identity/v3/roles/service => generated 85 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:02.211964 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b71f6d9-d29e-4440-8fb1-7e8740e57135 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:02.271688 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2779d5fc-4701-441e-aa3d-e9f14e64a187 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:02.271975 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2779d5fc-4701-441e-aa3d-e9f14e64a187 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:02.272262 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2779d5fc-4701-441e-aa3d-e9f14e64a187 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:02.273789 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 273/545] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:02.279070 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb195c3-5e46-4bed-9687-f499e97dbfb1 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:02.279358 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb195c3-5e46-4bed-9687-f499e97dbfb1 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:02.279639 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eb195c3-5e46-4bed-9687-f499e97dbfb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:02.308718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-5eb195c3-5e46-4bed-9687-f499e97dbfb1 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:02.311558 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-5eb195c3-5e46-4bed-9687-f499e97dbfb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:02.404919 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b71f6d9-d29e-4440-8fb1-7e8740e57135 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:02.445417 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 274/546] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 12 17:20:02.466760 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb195c3-5e46-4bed-9687-f499e97dbfb1 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:02.467666 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b54b297-12bf-4500-9242-791dadca69ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:02.468543 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b54b297-12bf-4500-9242-791dadca69ef None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:02.469607 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b54b297-12bf-4500-9242-791dadca69ef None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:02.470082 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b54b297-12bf-4500-9242-791dadca69ef None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:02.471388 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b54b297-12bf-4500-9242-791dadca69ef None admin] RBAC: Adding query filter params (name=service) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:02.471912 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b54b297-12bf-4500-9242-791dadca69ef None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:02.473490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b54b297-12bf-4500-9242-791dadca69ef None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:02.482914 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 273/547] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Jul 12 17:20:02 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:02.491323 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b54b297-12bf-4500-9242-791dadca69ef None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:02.512271 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:02.636628 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:02.642215 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eb195c3-5e46-4bed-9687-f499e97dbfb1 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:02.657173 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b54b297-12bf-4500-9242-791dadca69ef None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:02.692733 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a74a55a-f70d-4a5f-8858-51e7329c9e25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:02.694091 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a74a55a-f70d-4a5f-8858-51e7329c9e25 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:02.694620 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a74a55a-f70d-4a5f-8858-51e7329c9e25 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:02.695098 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a74a55a-f70d-4a5f-8858-51e7329c9e25 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:02.706676 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a74a55a-f70d-4a5f-8858-51e7329c9e25 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:20:02.714981 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-7a74a55a-f70d-4a5f-8858-51e7329c9e25 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:20:02.714981 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:02.714981 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:02.714981 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:02.714981 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:02.714981 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:02.714981 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:02.714981 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:02.714981 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:02.714981 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:02.714981 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:02.714981 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:02.714981 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:20:02.714981 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:20:02.714981 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:20:02.714981 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:20:02.715852 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:20:02.715852 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:20:02.715852 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:20:02.715852 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:20:02.715852 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:02.715852 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:02.715852 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:02.715852 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:02.715852 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:02.715852 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:02.715852 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:02.715852 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:02.715852 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:02.715852 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:02.715852 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:02.715852 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:02.717133 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:02.717133 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:02.717133 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:02.717133 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:02.717133 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:20:02.717133 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:20:02.717133 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:20:02.717133 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:20:02.717133 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:20:02.717133 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:20:02.717133 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:20:02.717133 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:02.720817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 275/548] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Jul 12 17:20:02 2022] GET /identity/v3/roles/service => generated 85 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:20:02.726723 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a74a55a-f70d-4a5f-8858-51e7329c9e25 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:02.735583 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e6af33-e060-4ef7-97f2-3d1311753b38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:02.736406 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e6af33-e060-4ef7-97f2-3d1311753b38 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:02.736691 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e6af33-e060-4ef7-97f2-3d1311753b38 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:02.736969 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e6af33-e060-4ef7-97f2-3d1311753b38 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:02.760403 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e6af33-e060-4ef7-97f2-3d1311753b38 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:02.761870 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e6af33-e060-4ef7-97f2-3d1311753b38 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:02.790503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-d0e6af33-e060-4ef7-97f2-3d1311753b38 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:02.790503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:02.790503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:02.790503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:02.790503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:02.790503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:02.790503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:02.790503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:02.790503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:02.790503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:02.790503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:02.790503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:02.790503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:02.790503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:02.790503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:02.791432 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:02.791432 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:02.791432 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:02.791432 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:02.791432 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:02.791432 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:02.791432 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:02.791432 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:02.791432 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:02.791432 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:02.791432 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:02.791432 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:02.791432 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:02.791432 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:02.791432 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:02.791432 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:02.791432 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:02.792319 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:02.792319 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:02.792319 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:02.792319 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:02.798682 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 274/549] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 308 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:02.804806 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e6af33-e060-4ef7-97f2-3d1311753b38 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:02.889550 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a74a55a-f70d-4a5f-8858-51e7329c9e25 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:02.940173 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:02.963494 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63b0961-ec34-4e4d-94b6-c73e098e8e15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:02.964628 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63b0961-ec34-4e4d-94b6-c73e098e8e15 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:02.965186 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63b0961-ec34-4e4d-94b6-c73e098e8e15 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:02.965665 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63b0961-ec34-4e4d-94b6-c73e098e8e15 None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:02.966882 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a63b0961-ec34-4e4d-94b6-c73e098e8e15 None admin] RBAC: Adding query filter params (name=service) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:02.969840 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a63b0961-ec34-4e4d-94b6-c73e098e8e15 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:02.971473 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a63b0961-ec34-4e4d-94b6-c73e098e8e15 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:02.981152 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 276/550] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Jul 12 17:20:02 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:02.987859 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63b0961-ec34-4e4d-94b6-c73e098e8e15 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:03.027877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e6af33-e060-4ef7-97f2-3d1311753b38 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:03.041950 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:03.097233 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8455b29e-a577-4d12-b6f0-630602948876 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:03.098472 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8455b29e-a577-4d12-b6f0-630602948876 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:03.098965 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8455b29e-a577-4d12-b6f0-630602948876 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:03.099444 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8455b29e-a577-4d12-b6f0-630602948876 None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:03.100376 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8455b29e-a577-4d12-b6f0-630602948876 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:03.100903 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8455b29e-a577-4d12-b6f0-630602948876 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:03.102509 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8455b29e-a577-4d12-b6f0-630602948876 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:03.117414 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 275/551] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 313 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:03.124192 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8455b29e-a577-4d12-b6f0-630602948876 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:03.191590 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63b0961-ec34-4e4d-94b6-c73e098e8e15 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:03.247122 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7d2e60-1b38-4cea-9943-93bd7d223ceb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:03.247926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7d2e60-1b38-4cea-9943-93bd7d223ceb None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:03.248215 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7d2e60-1b38-4cea-9943-93bd7d223ceb None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:03.248494 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7d2e60-1b38-4cea-9943-93bd7d223ceb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:03.261234 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8455b29e-a577-4d12-b6f0-630602948876 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:03.273429 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7d2e60-1b38-4cea-9943-93bd7d223ceb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:03.273429 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7d2e60-1b38-4cea-9943-93bd7d223ceb None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:03.288105 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:03.299196 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-bc7d2e60-1b38-4cea-9943-93bd7d223ceb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:03.299196 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:03.299196 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:03.299196 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:03.299196 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:03.299196 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:03.299196 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:03.299196 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:03.299196 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:03.299196 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:03.299196 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:03.299196 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:03.299196 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:03.299196 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:03.299196 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:03.301773 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:03.301773 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:03.301773 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:03.301773 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:03.301773 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:03.301773 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:03.301773 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:03.301773 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:03.301773 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:03.301773 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:03.301773 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:03.301773 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:03.301773 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:03.301773 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:03.301773 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:03.301773 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:03.301773 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:03.307644 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:03.307644 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:03.307644 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:03.307644 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:03.307862 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3668a34e-3265-4358-8405-2bc1ba6ab72d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:03.307862 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3668a34e-3265-4358-8405-2bc1ba6ab72d None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:03.307862 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3668a34e-3265-4358-8405-2bc1ba6ab72d None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:03.307862 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3668a34e-3265-4358-8405-2bc1ba6ab72d None admin] PATH_INFO: `/v3/users/swift` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:03.308219 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 277/552] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 314 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:20:03.309070 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230258c3-d80a-4dee-988e-87dd6cecbb09 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:03.309358 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230258c3-d80a-4dee-988e-87dd6cecbb09 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:03.309660 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230258c3-d80a-4dee-988e-87dd6cecbb09 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:03.311414 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 278/553] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:03.312119 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230258c3-d80a-4dee-988e-87dd6cecbb09 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:03.316169 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3668a34e-3265-4358-8405-2bc1ba6ab72d None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:03.317862 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3668a34e-3265-4358-8405-2bc1ba6ab72d None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:03.335450 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-3668a34e-3265-4358-8405-2bc1ba6ab72d None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 12 17:20:03.335450 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:03.335450 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:03.335450 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:03.335450 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:03.335450 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:03.335450 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:03.335450 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:03.335450 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:03.335450 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:03.335450 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:03.335450 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:03.335450 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:20:03.335450 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:20:03.335450 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:20:03.335450 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:20:03.336431 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:03.336431 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:03.336431 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:20:03.336431 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:03.336431 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:20:03.336431 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:03.336431 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:03.336431 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:03.336431 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:03.336431 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:03.336431 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:03.336431 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:03.336431 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:03.336431 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:03.336431 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:03.336431 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:03.336431 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:03.336431 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:03.337374 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:03.337374 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:03.337374 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:20:03.337374 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:20:03.337374 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:20:03.337374 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:20:03.337374 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:20:03.337374 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:20:03.337374 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 12 17:20:03.337374 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:03.340078 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 276/554] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Jul 12 17:20:03 2022] GET /identity/v3/users/swift => generated 83 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:03.347048 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f17d04f-9140-463a-8eb0-96daff2bc54a None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:03.347609 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f17d04f-9140-463a-8eb0-96daff2bc54a None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:03.348092 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f17d04f-9140-463a-8eb0-96daff2bc54a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:03.367309 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-8f17d04f-9140-463a-8eb0-96daff2bc54a None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:03.369124 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-8f17d04f-9140-463a-8eb0-96daff2bc54a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:03.398388 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:03.425614 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230258c3-d80a-4dee-988e-87dd6cecbb09 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:03.447822 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 277/555] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:20:03.449096 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f17d04f-9140-463a-8eb0-96daff2bc54a None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:03.481886 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-751de5df-0939-42d1-975f-84ffdb290f1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:03.482863 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751de5df-0939-42d1-975f-84ffdb290f1c None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:03.482976 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751de5df-0939-42d1-975f-84ffdb290f1c None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:03.483229 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751de5df-0939-42d1-975f-84ffdb290f1c None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:03.483984 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-751de5df-0939-42d1-975f-84ffdb290f1c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:03.484298 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-751de5df-0939-42d1-975f-84ffdb290f1c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:03.486646 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-751de5df-0939-42d1-975f-84ffdb290f1c None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:03.505159 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 279/556] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:03.510190 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-751de5df-0939-42d1-975f-84ffdb290f1c None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:03.569159 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f17d04f-9140-463a-8eb0-96daff2bc54a None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:03.584741 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:03.609081 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d8ddad-e83a-4af9-b875-e09629610a18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:03.610912 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d8ddad-e83a-4af9-b875-e09629610a18 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:03.611407 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d8ddad-e83a-4af9-b875-e09629610a18 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:03.611901 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d8ddad-e83a-4af9-b875-e09629610a18 None admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:03.613109 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89d8ddad-e83a-4af9-b875-e09629610a18 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:03.613641 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89d8ddad-e83a-4af9-b875-e09629610a18 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:03.615420 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89d8ddad-e83a-4af9-b875-e09629610a18 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:03.622831 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-751de5df-0939-42d1-975f-84ffdb290f1c None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:03.642450 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 278/557] 10.222.0.16 () {56 vars in 1135 bytes} [Tue Jul 12 17:20:03 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:03.643735 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d8ddad-e83a-4af9-b875-e09629610a18 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:03.663126 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6577288a-97e3-483f-8e69-319712058d69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:03.664030 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6577288a-97e3-483f-8e69-319712058d69 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:03.664325 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6577288a-97e3-483f-8e69-319712058d69 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:03.664604 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6577288a-97e3-483f-8e69-319712058d69 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:03.666742 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:03.678814 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6577288a-97e3-483f-8e69-319712058d69 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:20:03.687096 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-6577288a-97e3-483f-8e69-319712058d69 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:20:03.687096 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:03.687096 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:03.687096 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:03.687096 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:03.687096 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:03.687096 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:03.687096 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:03.687096 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:03.687096 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:03.687096 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:03.687096 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:03.687096 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:20:03.687096 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:20:03.687096 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:20:03.687096 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:20:03.689114 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:20:03.689114 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:20:03.689114 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:20:03.689114 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:20:03.689114 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:03.689114 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:03.689114 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:03.689114 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:03.689114 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:03.689114 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:03.689114 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:03.689114 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:03.689114 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:03.689114 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:03.689114 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:03.689114 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:03.690056 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:03.690056 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:03.690056 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:03.690056 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:03.690056 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:20:03.690056 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:20:03.690056 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:20:03.690056 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:20:03.690056 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:20:03.690056 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:20:03.690056 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:20:03.690056 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:03.693978 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 280/558] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Jul 12 17:20:03 2022] GET /identity/v3/roles/service => generated 85 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:03.694930 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6577288a-97e3-483f-8e69-319712058d69 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:03.758241 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d8ddad-e83a-4af9-b875-e09629610a18 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:03.795192 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0daffca-d920-4051-8bb6-db4a59105427 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:03.796410 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0daffca-d920-4051-8bb6-db4a59105427 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:03.797500 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0daffca-d920-4051-8bb6-db4a59105427 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:03.798005 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0daffca-d920-4051-8bb6-db4a59105427 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:03.806318 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0daffca-d920-4051-8bb6-db4a59105427 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:03.808157 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0daffca-d920-4051-8bb6-db4a59105427 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:03.811473 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6577288a-97e3-483f-8e69-319712058d69 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:03.824537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-c0daffca-d920-4051-8bb6-db4a59105427 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 12 17:20:03.824537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:03.824537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:03.824537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:03.824537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:03.824537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:03.824537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:03.824537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:03.824537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:03.824537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:03.824537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:03.824537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:03.824537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:20:03.824537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:20:03.824537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:20:03.824537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:20:03.825556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:03.825556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:03.825556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:20:03.825556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:03.825556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:20:03.825556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:03.825556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:03.825556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:03.825556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:03.825556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:03.825556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:03.825556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:03.825556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:03.825556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:03.825556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:03.825556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:03.825556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:03.825556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:03.826624 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:03.826624 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:03.826624 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:20:03.826624 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:20:03.826624 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:20:03.826624 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:20:03.826624 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:20:03.826624 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:20:03.826624 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 12 17:20:03.826624 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:03.834271 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 279/559] 10.222.0.16 () {56 vars in 1093 bytes} [Tue Jul 12 17:20:03 2022] GET /identity/v3/users/nova => generated 82 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:03.835429 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0daffca-d920-4051-8bb6-db4a59105427 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:03.850373 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6410dc06-abf8-4066-aeef-8b16faa68436 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:03.851159 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6410dc06-abf8-4066-aeef-8b16faa68436 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:03.855296 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6410dc06-abf8-4066-aeef-8b16faa68436 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:03.855296 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6410dc06-abf8-4066-aeef-8b16faa68436 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:03.868124 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:03.923224 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:03.932395 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6410dc06-abf8-4066-aeef-8b16faa68436 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:03.933974 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6410dc06-abf8-4066-aeef-8b16faa68436 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:03.943003 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0daffca-d920-4051-8bb6-db4a59105427 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:03.979202 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e404bd-fad0-467d-8b69-d72d9c9644fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:03.979870 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e404bd-fad0-467d-8b69-d72d9c9644fb None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:03.980156 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e404bd-fad0-467d-8b69-d72d9c9644fb None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:03.980431 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e404bd-fad0-467d-8b69-d72d9c9644fb None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:03.981178 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e404bd-fad0-467d-8b69-d72d9c9644fb None admin] RBAC: Adding query filter params (name=service) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:03.981550 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e404bd-fad0-467d-8b69-d72d9c9644fb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:03.982878 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e404bd-fad0-467d-8b69-d72d9c9644fb None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:03.991345 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 280/560] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Jul 12 17:20:03 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:03.992120 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e404bd-fad0-467d-8b69-d72d9c9644fb None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:04.075214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 281/561] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:20:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 12 17:20:04.076453 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b93919-051f-475b-a9e9-93ff5eeb11dd None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:04.076741 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b93919-051f-475b-a9e9-93ff5eeb11dd None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:04.077209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b93919-051f-475b-a9e9-93ff5eeb11dd None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:04.082687 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 282/562] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:04 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:04.083967 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0e510d-7615-4b95-aa64-bfa948b23a07 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:04.084281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0e510d-7615-4b95-aa64-bfa948b23a07 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:04.084528 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0e510d-7615-4b95-aa64-bfa948b23a07 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:04.085790 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 283/563] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:04.086486 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0e510d-7615-4b95-aa64-bfa948b23a07 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:04.100175 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:04.109201 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e404bd-fad0-467d-8b69-d72d9c9644fb None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:04.144562 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-228c5779-78ec-41c6-a25f-04fc4a66ca9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:04.145216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228c5779-78ec-41c6-a25f-04fc4a66ca9d None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:04.145511 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228c5779-78ec-41c6-a25f-04fc4a66ca9d None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:04.145797 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228c5779-78ec-41c6-a25f-04fc4a66ca9d None admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:04.146675 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-228c5779-78ec-41c6-a25f-04fc4a66ca9d None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:04.146993 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-228c5779-78ec-41c6-a25f-04fc4a66ca9d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:04.148419 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-228c5779-78ec-41c6-a25f-04fc4a66ca9d None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:04.165324 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 281/564] 10.222.0.16 () {56 vars in 1133 bytes} [Tue Jul 12 17:20:03 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:04.166132 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-228c5779-78ec-41c6-a25f-04fc4a66ca9d None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:04.186836 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:04.195111 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0e510d-7615-4b95-aa64-bfa948b23a07 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:04.230132 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0bf8c36-653a-47b3-b041-b87c3ebbcb4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:04.230943 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0bf8c36-653a-47b3-b041-b87c3ebbcb4f None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:04.231232 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0bf8c36-653a-47b3-b041-b87c3ebbcb4f None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:04.231518 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0bf8c36-653a-47b3-b041-b87c3ebbcb4f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:04.243081 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0bf8c36-653a-47b3-b041-b87c3ebbcb4f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:04.244531 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0bf8c36-653a-47b3-b041-b87c3ebbcb4f None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:04.262931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-e0bf8c36-653a-47b3-b041-b87c3ebbcb4f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:04.262931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:04.262931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:04.262931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:04.262931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:04.262931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:04.262931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:04.262931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:04.262931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:04.262931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:04.262931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:04.262931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:04.262931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:04.262931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:04.262931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:04.264252 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:04.264252 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:04.264252 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:04.264252 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:04.264252 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:04.264252 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:04.264252 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:04.264252 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:04.264252 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:04.264252 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:04.264252 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:04.264252 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:04.264252 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:04.264252 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:04.264252 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:04.264252 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:04.264252 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:04.265713 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:04.265713 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:04.265713 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:04.265713 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:04.267772 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-228c5779-78ec-41c6-a25f-04fc4a66ca9d None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:04.269073 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 284/565] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:04.270480 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e6a2f5-cf1a-418d-9fba-0bb838806677 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:04.270761 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e6a2f5-cf1a-418d-9fba-0bb838806677 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:04.271046 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e6a2f5-cf1a-418d-9fba-0bb838806677 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:04.287405 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-97e6a2f5-cf1a-418d-9fba-0bb838806677 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:04.288628 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-97e6a2f5-cf1a-418d-9fba-0bb838806677 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:04.302135 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20da3837-347d-4dbe-bf1a-e6ab39384272 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:04.302967 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20da3837-347d-4dbe-bf1a-e6ab39384272 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:04.303256 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20da3837-347d-4dbe-bf1a-e6ab39384272 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:04.303541 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20da3837-347d-4dbe-bf1a-e6ab39384272 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:04.314379 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20da3837-347d-4dbe-bf1a-e6ab39384272 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:04.315847 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20da3837-347d-4dbe-bf1a-e6ab39384272 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:04.334249 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-20da3837-347d-4dbe-bf1a-e6ab39384272 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:04.334249 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:04.334249 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:04.334249 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:04.334249 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:04.334249 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:04.334249 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:04.334249 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:04.334249 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:04.334249 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:04.334249 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:04.334249 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:04.334249 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:04.334249 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:04.334249 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:04.335295 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:04.335295 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:04.335295 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:04.335295 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:04.335295 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:04.335295 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:04.335295 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:04.335295 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:04.335295 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:04.335295 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:04.335295 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:04.335295 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:04.335295 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:04.335295 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:04.335295 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:04.335295 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:04.335295 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:04.336332 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:04.336332 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:04.336332 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:04.336332 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:04.336332 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 282/566] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:04.336752 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa745892-90ff-4e15-bd05-c276dadcc5f1 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:04.337054 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa745892-90ff-4e15-bd05-c276dadcc5f1 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:04.337299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa745892-90ff-4e15-bd05-c276dadcc5f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:04.342122 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:04.355085 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-aa745892-90ff-4e15-bd05-c276dadcc5f1 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:04.356281 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-aa745892-90ff-4e15-bd05-c276dadcc5f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:04.359075 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 285/567] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:20:04.360304 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e6a2f5-cf1a-418d-9fba-0bb838806677 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:04.427666 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 283/568] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:20:04.427666 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa745892-90ff-4e15-bd05-c276dadcc5f1 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:04.439794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:04.463091 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e6a2f5-cf1a-418d-9fba-0bb838806677 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:04.498539 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ff7f3c-cbb4-4bf8-937a-1bb40867ec1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:04.499328 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ff7f3c-cbb4-4bf8-937a-1bb40867ec1f None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:04.499649 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ff7f3c-cbb4-4bf8-937a-1bb40867ec1f None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:04.499926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ff7f3c-cbb4-4bf8-937a-1bb40867ec1f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:04.509927 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ff7f3c-cbb4-4bf8-937a-1bb40867ec1f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:04.511409 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ff7f3c-cbb4-4bf8-937a-1bb40867ec1f None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:04.528118 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa745892-90ff-4e15-bd05-c276dadcc5f1 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:04.528439 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-43ff7f3c-cbb4-4bf8-937a-1bb40867ec1f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:04.528439 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:04.528439 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:04.528439 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:04.528439 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:04.528439 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:04.528439 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:04.528439 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:04.528439 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:04.528439 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:04.528439 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:04.528439 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:04.528439 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:04.528439 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:04.528439 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:04.529464 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:04.529464 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:04.529464 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:04.529464 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:04.529464 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:04.529464 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:04.529464 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:04.529464 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:04.529464 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:04.529464 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:04.529464 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:04.529464 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:04.529464 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:04.529464 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:04.529464 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:04.529464 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:04.529464 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:04.530366 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:04.530366 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:04.530366 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:04.530366 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:04.530366 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 286/569] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:04.530366 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ff7f3c-cbb4-4bf8-937a-1bb40867ec1f None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:04.559526 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:04.565108 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c25de7-b729-4c2b-8a94-b0192880e1f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:04.565780 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c25de7-b729-4c2b-8a94-b0192880e1f4 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:04.568873 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c25de7-b729-4c2b-8a94-b0192880e1f4 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:04.568873 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c25de7-b729-4c2b-8a94-b0192880e1f4 None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:04.568873 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c25de7-b729-4c2b-8a94-b0192880e1f4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:04.568873 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c25de7-b729-4c2b-8a94-b0192880e1f4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:04.568873 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c25de7-b729-4c2b-8a94-b0192880e1f4 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:04.580139 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 284/570] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:04.580991 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c25de7-b729-4c2b-8a94-b0192880e1f4 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:04.637021 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ff7f3c-cbb4-4bf8-937a-1bb40867ec1f None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:04.664289 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:04.675099 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08e1a5e-9065-4d78-bca5-517072e3044c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:04.675771 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08e1a5e-9065-4d78-bca5-517072e3044c None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:04.676060 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08e1a5e-9065-4d78-bca5-517072e3044c None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:04.682646 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08e1a5e-9065-4d78-bca5-517072e3044c None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:04.682646 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08e1a5e-9065-4d78-bca5-517072e3044c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:04.682646 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08e1a5e-9065-4d78-bca5-517072e3044c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:04.682646 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c08e1a5e-9065-4d78-bca5-517072e3044c None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:04.683566 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c25de7-b729-4c2b-8a94-b0192880e1f4 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:04.690295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 287/571] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:04.691519 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08e1a5e-9065-4d78-bca5-517072e3044c None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:04.721011 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14d0926-3773-49aa-a850-565600e17b41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:04.721804 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14d0926-3773-49aa-a850-565600e17b41 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:04.722093 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14d0926-3773-49aa-a850-565600e17b41 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:04.722368 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14d0926-3773-49aa-a850-565600e17b41 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:04.732443 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f14d0926-3773-49aa-a850-565600e17b41 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:20:04.741031 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-f14d0926-3773-49aa-a850-565600e17b41 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:20:04.741031 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:04.741031 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:04.741031 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:04.741031 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:04.741031 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:04.741031 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:04.741031 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:04.741031 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:04.741031 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:04.741031 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:04.741031 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:04.741031 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:20:04.741031 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:20:04.741031 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:20:04.741031 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:20:04.742019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:20:04.742019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:20:04.742019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:20:04.742019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:20:04.742019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:04.742019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:04.742019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:04.742019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:04.742019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:04.742019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:04.742019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:04.742019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:04.742019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:04.742019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:04.742019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:04.742019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:04.742918 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:04.742918 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:04.742918 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:04.742918 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:04.742918 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:20:04.742918 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:20:04.742918 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:20:04.742918 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:20:04.742918 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:20:04.742918 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:20:04.742918 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:20:04.742918 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:04.742918 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 285/572] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Jul 12 17:20:04 2022] GET /identity/v3/roles/service => generated 85 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:04.743774 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14d0926-3773-49aa-a850-565600e17b41 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:04.795022 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08e1a5e-9065-4d78-bca5-517072e3044c None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:04.800045 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:04.829400 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56686b07-2c1c-4bb5-a8af-038a5e091e70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:04.830218 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56686b07-2c1c-4bb5-a8af-038a5e091e70 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:04.830534 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56686b07-2c1c-4bb5-a8af-038a5e091e70 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:04.830812 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56686b07-2c1c-4bb5-a8af-038a5e091e70 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:04.840742 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56686b07-2c1c-4bb5-a8af-038a5e091e70 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:20:04.847978 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14d0926-3773-49aa-a850-565600e17b41 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:04.849193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-56686b07-2c1c-4bb5-a8af-038a5e091e70 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:20:04.849193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:04.849193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:04.849193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:04.849193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:04.849193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:04.849193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:04.849193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:04.849193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:04.849193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:04.849193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:04.849193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:04.849193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:20:04.849193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:20:04.849193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:20:04.849193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:20:04.850108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:20:04.850108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:20:04.850108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:20:04.850108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:20:04.850108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:04.850108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:04.850108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:04.850108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:04.850108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:04.850108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:04.850108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:04.850108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:04.850108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:04.850108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:04.850108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:04.850108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:04.851048 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:04.851048 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:04.851048 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:04.851048 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:04.851048 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:20:04.851048 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:20:04.851048 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:20:04.851048 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:20:04.851048 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:20:04.851048 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:20:04.851048 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:20:04.851048 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:04.851048 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 288/573] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Jul 12 17:20:04 2022] GET /identity/v3/roles/service => generated 85 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:04.851753 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56686b07-2c1c-4bb5-a8af-038a5e091e70 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:04.886258 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1cf6bc6-e173-4588-91d8-96bd02de192b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:04.886963 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cf6bc6-e173-4588-91d8-96bd02de192b None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:04.887307 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cf6bc6-e173-4588-91d8-96bd02de192b None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:04.887544 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cf6bc6-e173-4588-91d8-96bd02de192b None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:04.888282 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1cf6bc6-e173-4588-91d8-96bd02de192b None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:04.888598 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1cf6bc6-e173-4588-91d8-96bd02de192b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:04.889972 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1cf6bc6-e173-4588-91d8-96bd02de192b None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:04.902394 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 286/574] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:04.903218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1cf6bc6-e173-4588-91d8-96bd02de192b None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:04.904553 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:04.956230 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56686b07-2c1c-4bb5-a8af-038a5e091e70 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:04.990745 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91545166-0336-480e-8046-9d184b08e197 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:04.991539 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91545166-0336-480e-8046-9d184b08e197 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:04.991831 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91545166-0336-480e-8046-9d184b08e197 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:04.992117 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91545166-0336-480e-8046-9d184b08e197 None admin] PATH_INFO: `/v3/users/glance` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:04.999974 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91545166-0336-480e-8046-9d184b08e197 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:05.001422 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91545166-0336-480e-8046-9d184b08e197 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:05.009057 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1cf6bc6-e173-4588-91d8-96bd02de192b None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:05.016376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-91545166-0336-480e-8046-9d184b08e197 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Jul 12 17:20:05.016376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:05.016376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:05.016376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:05.016376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:05.016376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:05.016376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:05.016376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:05.016376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:05.016376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:05.016376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:05.016376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:05.016376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:20:05.016376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:20:05.016376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:20:05.016376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:20:05.017374 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:05.017374 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:05.017374 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:20:05.017374 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:05.017374 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:20:05.017374 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:05.017374 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:05.017374 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:05.017374 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:05.017374 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:05.017374 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:05.017374 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:05.017374 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:05.017374 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:05.017374 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:05.017374 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:05.017374 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:05.017374 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:05.018862 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:05.018862 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:05.018862 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:20:05.018862 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:20:05.018862 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:20:05.018862 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:20:05.018862 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:20:05.018862 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:20:05.018862 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Jul 12 17:20:05.018862 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:05.018862 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 289/575] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:20:04 2022] GET /identity/v3/users/glance => generated 84 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:05.018862 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91545166-0336-480e-8046-9d184b08e197 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:05.024272 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:05.046885 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f048c8b-31c1-49ac-8e74-f9af515b7011 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:05.047687 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f048c8b-31c1-49ac-8e74-f9af515b7011 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:05.047974 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f048c8b-31c1-49ac-8e74-f9af515b7011 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:05.048259 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f048c8b-31c1-49ac-8e74-f9af515b7011 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:05.055358 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f048c8b-31c1-49ac-8e74-f9af515b7011 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:05.057117 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f048c8b-31c1-49ac-8e74-f9af515b7011 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:05.071019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-5f048c8b-31c1-49ac-8e74-f9af515b7011 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:05.071019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:05.071019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:05.071019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:05.071019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:05.071019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:05.071019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:05.071019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:05.071019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:05.071019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:05.071019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:05.071019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:05.071019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:20:05.071019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:20:05.071019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:20:05.071945 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:20:05.071945 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:05.071945 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:05.071945 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:05.071945 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:05.071945 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:05.071945 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:05.071945 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:05.071945 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:05.071945 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:05.071945 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:05.071945 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:05.071945 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:05.071945 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:05.071945 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:05.071945 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:05.071945 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:05.073456 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:20:05.073456 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:20:05.073456 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:20:05.073456 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:20:05.073456 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:20:05.073456 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:20:05.073456 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:05.073456 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:05.073456 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 287/576] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:20:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:05.073456 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f048c8b-31c1-49ac-8e74-f9af515b7011 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:05.127678 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91545166-0336-480e-8046-9d184b08e197 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:05.136628 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:05.161392 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749da3da-a6f5-4a92-aa05-6ae1890b5bc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:05.161980 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749da3da-a6f5-4a92-aa05-6ae1890b5bc7 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:05.162269 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749da3da-a6f5-4a92-aa05-6ae1890b5bc7 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:05.162580 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749da3da-a6f5-4a92-aa05-6ae1890b5bc7 None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:05.163331 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749da3da-a6f5-4a92-aa05-6ae1890b5bc7 None admin] RBAC: Adding query filter params (name=service) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:05.163639 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749da3da-a6f5-4a92-aa05-6ae1890b5bc7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:05.165077 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749da3da-a6f5-4a92-aa05-6ae1890b5bc7 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:05.174154 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 290/577] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Jul 12 17:20:05 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:05.175031 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f048c8b-31c1-49ac-8e74-f9af515b7011 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:05.175143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749da3da-a6f5-4a92-aa05-6ae1890b5bc7 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:05.211409 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6da3b86-459c-4ca2-93dd-7ef44d17005f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:05.212079 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6da3b86-459c-4ca2-93dd-7ef44d17005f None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:05.212368 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6da3b86-459c-4ca2-93dd-7ef44d17005f None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:05.212649 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6da3b86-459c-4ca2-93dd-7ef44d17005f None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:05.213593 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6da3b86-459c-4ca2-93dd-7ef44d17005f None admin] RBAC: Adding query filter params (name=service) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:05.213767 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6da3b86-459c-4ca2-93dd-7ef44d17005f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:05.215167 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6da3b86-459c-4ca2-93dd-7ef44d17005f None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:05.223953 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 288/578] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Jul 12 17:20:05 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:05.224810 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6da3b86-459c-4ca2-93dd-7ef44d17005f None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:05.259977 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:05.275784 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749da3da-a6f5-4a92-aa05-6ae1890b5bc7 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:05.310973 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f1cb0b7-694c-4acc-b448-4ae895c6dc92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:05.311764 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1cb0b7-694c-4acc-b448-4ae895c6dc92 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:05.312049 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1cb0b7-694c-4acc-b448-4ae895c6dc92 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:05.312370 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1cb0b7-694c-4acc-b448-4ae895c6dc92 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:05.319679 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f1cb0b7-694c-4acc-b448-4ae895c6dc92 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:05.321161 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f1cb0b7-694c-4acc-b448-4ae895c6dc92 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:05.325844 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6da3b86-459c-4ca2-93dd-7ef44d17005f None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:05.334993 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-8f1cb0b7-694c-4acc-b448-4ae895c6dc92 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:05.334993 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:05.334993 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:05.334993 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:05.334993 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:05.334993 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:05.334993 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:05.334993 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:05.334993 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:05.334993 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:05.334993 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:05.334993 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:05.334993 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:20:05.334993 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:20:05.334993 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:20:05.336003 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:20:05.336003 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:05.336003 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:05.336003 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:05.336003 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:05.336003 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:05.336003 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:05.336003 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:05.336003 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:05.336003 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:05.336003 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:05.336003 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:05.336003 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:05.336003 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:05.336003 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:05.336003 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:05.336003 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:05.336900 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:20:05.336900 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:20:05.336900 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:20:05.336900 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:20:05.336900 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:20:05.336900 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:20:05.336900 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:05.336900 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:05.336900 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 291/579] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:20:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:05.338721 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f1cb0b7-694c-4acc-b448-4ae895c6dc92 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:05.360763 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff654eb6-8ba2-4682-acea-f202ac4edac4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:05.361429 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff654eb6-8ba2-4682-acea-f202ac4edac4 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:05.361717 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff654eb6-8ba2-4682-acea-f202ac4edac4 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:05.361996 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff654eb6-8ba2-4682-acea-f202ac4edac4 None admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:05.362877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff654eb6-8ba2-4682-acea-f202ac4edac4 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:05.363191 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff654eb6-8ba2-4682-acea-f202ac4edac4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:05.364623 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff654eb6-8ba2-4682-acea-f202ac4edac4 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:05.370670 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:05.381948 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 289/580] 10.222.0.16 () {56 vars in 1137 bytes} [Tue Jul 12 17:20:05 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:05.382814 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff654eb6-8ba2-4682-acea-f202ac4edac4 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:05.440883 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f1cb0b7-694c-4acc-b448-4ae895c6dc92 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:05.471619 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:05.475893 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7b0649-b829-4124-a857-298485d93fc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:05.476556 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7b0649-b829-4124-a857-298485d93fc2 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:05.476844 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7b0649-b829-4124-a857-298485d93fc2 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:05.477124 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7b0649-b829-4124-a857-298485d93fc2 None admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:05.477914 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a7b0649-b829-4124-a857-298485d93fc2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:05.478233 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a7b0649-b829-4124-a857-298485d93fc2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:05.479936 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a7b0649-b829-4124-a857-298485d93fc2 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:05.480100 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff654eb6-8ba2-4682-acea-f202ac4edac4 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:05.492129 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 292/581] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Jul 12 17:20:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:05.493016 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7b0649-b829-4124-a857-298485d93fc2 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:05.514889 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a473420-e277-4061-9343-20445c041dda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:05.515703 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a473420-e277-4061-9343-20445c041dda None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:05.516179 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a473420-e277-4061-9343-20445c041dda None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:05.516470 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a473420-e277-4061-9343-20445c041dda None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:05.526689 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a473420-e277-4061-9343-20445c041dda None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:05.528149 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a473420-e277-4061-9343-20445c041dda None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:05.545961 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-1a473420-e277-4061-9343-20445c041dda None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:05.545961 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:05.545961 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:05.545961 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:05.545961 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:05.545961 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:05.545961 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:05.545961 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:05.545961 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:05.545961 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:05.545961 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:05.545961 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:05.545961 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:05.545961 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:05.545961 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:05.547617 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:05.547617 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:05.547617 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:05.547617 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:05.547617 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:05.547617 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:05.547617 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:05.547617 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:05.547617 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:05.547617 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:05.547617 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:05.547617 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:05.547617 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:05.547617 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:05.547617 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:05.547617 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:05.547617 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:05.548496 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:05.548496 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:05.548496 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:05.548496 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:05.548496 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 290/582] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:05.549248 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a473420-e277-4061-9343-20445c041dda None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:05.591820 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:05.601734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7b0649-b829-4124-a857-298485d93fc2 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:05.635229 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4214bbd2-775b-4a84-9246-bca428e36a64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:05.636034 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4214bbd2-775b-4a84-9246-bca428e36a64 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:05.636318 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4214bbd2-775b-4a84-9246-bca428e36a64 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:05.636598 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4214bbd2-775b-4a84-9246-bca428e36a64 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:05.648665 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4214bbd2-775b-4a84-9246-bca428e36a64 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:05.650174 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4214bbd2-775b-4a84-9246-bca428e36a64 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:05.651023 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a473420-e277-4061-9343-20445c041dda None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:05.669145 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-4214bbd2-775b-4a84-9246-bca428e36a64 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:05.669145 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:05.669145 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:05.669145 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:05.669145 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:05.669145 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:05.669145 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:05.669145 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:05.669145 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:05.669145 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:05.669145 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:05.669145 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:05.669145 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:05.669145 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:05.669145 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:05.670240 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:05.670240 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:05.670240 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:05.670240 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:05.670240 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:05.670240 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:05.670240 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:05.670240 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:05.670240 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:05.670240 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:05.670240 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:05.670240 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:05.670240 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:05.670240 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:05.670240 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:05.670240 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:05.670240 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:05.671216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:05.671216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:05.671216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:05.671216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:05.671216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 293/583] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:05.671745 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4214bbd2-775b-4a84-9246-bca428e36a64 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:05.685823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e8ccf8-75c6-47f0-94cc-ea6b0e0bb0ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:05.686526 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e8ccf8-75c6-47f0-94cc-ea6b0e0bb0ae None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:05.686874 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e8ccf8-75c6-47f0-94cc-ea6b0e0bb0ae None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:05.687161 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e8ccf8-75c6-47f0-94cc-ea6b0e0bb0ae None admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:05.687991 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e8ccf8-75c6-47f0-94cc-ea6b0e0bb0ae None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:05.688307 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e8ccf8-75c6-47f0-94cc-ea6b0e0bb0ae None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:05.689742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e8ccf8-75c6-47f0-94cc-ea6b0e0bb0ae None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:05.701166 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 291/584] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Jul 12 17:20:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:05.701897 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e8ccf8-75c6-47f0-94cc-ea6b0e0bb0ae None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:05.705779 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:05.773030 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4214bbd2-775b-4a84-9246-bca428e36a64 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:05.806852 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce48406f-f357-4014-8b34-a4de8aaff00a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:05.807635 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce48406f-f357-4014-8b34-a4de8aaff00a None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:05.807919 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce48406f-f357-4014-8b34-a4de8aaff00a None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:05.808197 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce48406f-f357-4014-8b34-a4de8aaff00a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:05.809092 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e8ccf8-75c6-47f0-94cc-ea6b0e0bb0ae None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:05.818818 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce48406f-f357-4014-8b34-a4de8aaff00a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:05.820262 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce48406f-f357-4014-8b34-a4de8aaff00a None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:05.822004 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:05.839293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-ce48406f-f357-4014-8b34-a4de8aaff00a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:05.839293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:05.839293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:05.839293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:05.839293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:05.839293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:05.839293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:05.839293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:05.839293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:05.839293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:05.839293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:05.839293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:05.839293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:05.839293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:05.839293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:05.840265 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:05.840265 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:05.840265 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:05.840265 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:05.840265 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:05.840265 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:05.840265 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:05.840265 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:05.840265 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:05.840265 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:05.840265 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:05.840265 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:05.840265 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:05.840265 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:05.840265 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:05.840265 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:05.840265 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:05.841192 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:05.841192 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:05.841192 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:05.841192 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:05.841192 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 294/585] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:05.841887 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce48406f-f357-4014-8b34-a4de8aaff00a None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:05.847022 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b284f213-30e2-4f46-b61a-a3acbdea2154 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:05.847847 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b284f213-30e2-4f46-b61a-a3acbdea2154 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:05.848139 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b284f213-30e2-4f46-b61a-a3acbdea2154 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:05.848424 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b284f213-30e2-4f46-b61a-a3acbdea2154 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:05.849259 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b284f213-30e2-4f46-b61a-a3acbdea2154 None admin] RBAC: Adding query filter params (role.id=bef74d4975164f75bcec96ac47afea73, scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2, user.id=d7906d1a19c642bd93507a27bf3e3dae) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:05.849578 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b284f213-30e2-4f46-b61a-a3acbdea2154 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:05.851119 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b284f213-30e2-4f46-b61a-a3acbdea2154 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:05.857210 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 292/586] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:20:05 2022] GET /identity/v3/role_assignments?user.id=d7906d1a19c642bd93507a27bf3e3dae&scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2&role.id=bef74d4975164f75bcec96ac47afea73 => generated 259 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:05.857999 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b284f213-30e2-4f46-b61a-a3acbdea2154 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:05.939697 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:05.967895 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce48406f-f357-4014-8b34-a4de8aaff00a None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:05.981064 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b284f213-30e2-4f46-b61a-a3acbdea2154 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:06.017211 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271cfe8d-d23b-41c7-b4b8-ce3e85c40d08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:06.017754 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271cfe8d-d23b-41c7-b4b8-ce3e85c40d08 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:06.018044 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271cfe8d-d23b-41c7-b4b8-ce3e85c40d08 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:06.018318 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271cfe8d-d23b-41c7-b4b8-ce3e85c40d08 None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:06.019069 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-271cfe8d-d23b-41c7-b4b8-ce3e85c40d08 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:06.019380 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-271cfe8d-d23b-41c7-b4b8-ce3e85c40d08 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:06.020757 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-271cfe8d-d23b-41c7-b4b8-ce3e85c40d08 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:06.021821 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c98be6-bf7c-44ac-a75c-2fa2e8430c27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:06.022604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c98be6-bf7c-44ac-a75c-2fa2e8430c27 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:06.022822 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c98be6-bf7c-44ac-a75c-2fa2e8430c27 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:06.023108 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c98be6-bf7c-44ac-a75c-2fa2e8430c27 None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:06.023843 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c98be6-bf7c-44ac-a75c-2fa2e8430c27 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:06.024158 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c98be6-bf7c-44ac-a75c-2fa2e8430c27 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:06.025530 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c98be6-bf7c-44ac-a75c-2fa2e8430c27 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:06.037851 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 295/587] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:06.037851 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271cfe8d-d23b-41c7-b4b8-ce3e85c40d08 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:06.039383 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 293/588] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:06.043726 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c98be6-bf7c-44ac-a75c-2fa2e8430c27 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:06.102902 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:06.150447 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271cfe8d-d23b-41c7-b4b8-ce3e85c40d08 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:06.151373 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c98be6-bf7c-44ac-a75c-2fa2e8430c27 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:06.182826 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:06.188449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af50f21c-b06e-4d65-9a4e-3a70e4e4127f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:06.189372 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af50f21c-b06e-4d65-9a4e-3a70e4e4127f None admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:06.189658 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af50f21c-b06e-4d65-9a4e-3a70e4e4127f None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:06.189981 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af50f21c-b06e-4d65-9a4e-3a70e4e4127f None admin] PATH_INFO: `/v3/projects/fd4ed2fe037c4fb7ba49c724eb9414c2/users/9012d5b88f554dc596582a7693c2a72a/roles/bef74d4975164f75bcec96ac47afea73` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:06.191131 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04929cab-b248-42f8-894c-394f61a84adc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:06.191819 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04929cab-b248-42f8-894c-394f61a84adc None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:06.192105 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04929cab-b248-42f8-894c-394f61a84adc None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:06.192507 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04929cab-b248-42f8-894c-394f61a84adc None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:06.193358 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04929cab-b248-42f8-894c-394f61a84adc None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:06.193668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04929cab-b248-42f8-894c-394f61a84adc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:06.195070 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04929cab-b248-42f8-894c-394f61a84adc None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:06.205577 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af50f21c-b06e-4d65-9a4e-3a70e4e4127f None admin] RBAC: Authorizing `identity:create_grant(project_id=fd4ed2fe037c4fb7ba49c724eb9414c2, user_id=9012d5b88f554dc596582a7693c2a72a, role_id=bef74d4975164f75bcec96ac47afea73)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:06.207059 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af50f21c-b06e-4d65-9a4e-3a70e4e4127f None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:06.210756 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 294/589] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:06.211959 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04929cab-b248-42f8-894c-394f61a84adc None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:06.299457 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 296/590] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Jul 12 17:20:06 2022] PUT /identity/v3/projects/fd4ed2fe037c4fb7ba49c724eb9414c2/users/9012d5b88f554dc596582a7693c2a72a/roles/bef74d4975164f75bcec96ac47afea73 => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:20:06.300447 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af50f21c-b06e-4d65-9a4e-3a70e4e4127f None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:06.338349 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04929cab-b248-42f8-894c-394f61a84adc None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:06.382364 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0b4daa-e326-4c58-9df8-97e9e7ad4a10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:06.383366 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0b4daa-e326-4c58-9df8-97e9e7ad4a10 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:06.383721 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0b4daa-e326-4c58-9df8-97e9e7ad4a10 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:06.384004 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0b4daa-e326-4c58-9df8-97e9e7ad4a10 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:06.394597 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0b4daa-e326-4c58-9df8-97e9e7ad4a10 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:06.395763 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0b4daa-e326-4c58-9df8-97e9e7ad4a10 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:06.413294 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-1e0b4daa-e326-4c58-9df8-97e9e7ad4a10 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Jul 12 17:20:06.413294 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:06.413294 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:06.413294 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:06.413294 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:06.413294 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:06.413294 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:06.413294 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:06.413294 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:06.413294 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:06.413294 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:06.413294 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:06.413294 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:20:06.413294 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:20:06.413294 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:20:06.413294 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:20:06.414372 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:06.414372 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:06.414372 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:20:06.414372 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:06.414372 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:20:06.414372 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:06.414372 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:06.414372 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:06.414372 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:06.414372 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:06.414372 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:06.414372 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:06.414372 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:06.414372 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:06.414372 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:06.414372 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:06.414372 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:06.414372 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:06.415441 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:06.415441 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:06.415441 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:20:06.415441 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:20:06.415441 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:20:06.415441 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:20:06.415441 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:20:06.415441 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:20:06.415441 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Jul 12 17:20:06.415441 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:06.420248 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 295/591] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Jul 12 17:20:06 2022] GET /identity/v3/users/neutron => generated 85 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:06.420557 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0b4daa-e326-4c58-9df8-97e9e7ad4a10 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:06.462366 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af50f21c-b06e-4d65-9a4e-3a70e4e4127f None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:06.469318 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:06.487803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:06.507146 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-562ccb53-6b5b-4c2a-bc60-92eb3f733b27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:06.508511 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562ccb53-6b5b-4c2a-bc60-92eb3f733b27 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:06.508976 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562ccb53-6b5b-4c2a-bc60-92eb3f733b27 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:06.509426 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562ccb53-6b5b-4c2a-bc60-92eb3f733b27 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:06.520278 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-562ccb53-6b5b-4c2a-bc60-92eb3f733b27 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:06.521970 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-562ccb53-6b5b-4c2a-bc60-92eb3f733b27 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:06.543558 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-562ccb53-6b5b-4c2a-bc60-92eb3f733b27 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jul 12 17:20:06.543558 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:06.543558 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:06.543558 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:06.543558 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:06.543558 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:06.543558 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:06.543558 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:06.543558 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:06.543558 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:06.543558 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:06.543558 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:06.543558 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:20:06.543558 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:20:06.543558 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:20:06.543558 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:20:06.544776 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:06.544776 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:06.544776 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:20:06.544776 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:06.544776 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:20:06.544776 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:06.544776 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:06.544776 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:06.544776 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:06.544776 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:06.544776 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:06.544776 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:06.544776 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:06.544776 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:06.544776 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:06.544776 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:06.544776 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:06.544776 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:06.545786 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:06.545786 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:06.545786 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:20:06.545786 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:20:06.545786 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:20:06.545786 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:20:06.545786 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:20:06.545786 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:20:06.545786 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jul 12 17:20:06.545786 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:06.548995 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 297/592] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:20:06 2022] GET /identity/v3/users/cinder => generated 84 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:06.552933 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-562ccb53-6b5b-4c2a-bc60-92eb3f733b27 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:06.560525 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0b4daa-e326-4c58-9df8-97e9e7ad4a10 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:06.611806 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8bf2e9-ec73-4b5c-9ee6-7d44dec0465d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:06.612987 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8bf2e9-ec73-4b5c-9ee6-7d44dec0465d None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:06.613363 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8bf2e9-ec73-4b5c-9ee6-7d44dec0465d None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:06.614050 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba8bf2e9-ec73-4b5c-9ee6-7d44dec0465d None admin] PATH_INFO: `/v3/projects/service` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:06.622971 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba8bf2e9-ec73-4b5c-9ee6-7d44dec0465d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:06.625004 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba8bf2e9-ec73-4b5c-9ee6-7d44dec0465d None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:06.646093 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-ba8bf2e9-ec73-4b5c-9ee6-7d44dec0465d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:06.646093 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:06.646093 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:06.646093 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:06.646093 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:06.646093 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:06.646093 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:06.646093 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:06.646093 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:06.646093 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:06.646093 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:06.646093 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:06.646093 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:20:06.646093 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:20:06.646093 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:20:06.647115 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:20:06.647115 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:06.647115 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:06.647115 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:06.647115 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:06.647115 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:06.647115 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:06.647115 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:06.647115 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:06.647115 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:06.647115 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:06.647115 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:06.647115 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:06.647115 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:06.647115 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:06.647115 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:06.647115 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:06.648024 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:20:06.648024 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:20:06.648024 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:20:06.648024 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:20:06.648024 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:20:06.648024 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:20:06.648024 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:06.648024 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:06.650512 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 296/593] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:20:06 2022] GET /identity/v3/projects/service => generated 88 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:06.651556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8bf2e9-ec73-4b5c-9ee6-7d44dec0465d None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:06.680008 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-562ccb53-6b5b-4c2a-bc60-92eb3f733b27 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:06.726582 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7edb2375-3702-4fae-a134-9c34c8e1c5a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:06.728175 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edb2375-3702-4fae-a134-9c34c8e1c5a4 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:06.728998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edb2375-3702-4fae-a134-9c34c8e1c5a4 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:06.729421 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edb2375-3702-4fae-a134-9c34c8e1c5a4 None admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:06.731097 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7edb2375-3702-4fae-a134-9c34c8e1c5a4 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:06.731584 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7edb2375-3702-4fae-a134-9c34c8e1c5a4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:06.733682 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7edb2375-3702-4fae-a134-9c34c8e1c5a4 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:06.758036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 298/594] 10.222.0.16 () {56 vars in 1139 bytes} [Tue Jul 12 17:20:06 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:06.759149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7edb2375-3702-4fae-a134-9c34c8e1c5a4 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:06.789452 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba8bf2e9-ec73-4b5c-9ee6-7d44dec0465d None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:06.790332 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:06.805681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:06.833269 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97afaeae-78b1-4b41-a808-8a1ca79729c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:06.833948 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97afaeae-78b1-4b41-a808-8a1ca79729c3 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:06.834241 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97afaeae-78b1-4b41-a808-8a1ca79729c3 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:06.834576 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97afaeae-78b1-4b41-a808-8a1ca79729c3 None admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:06.835434 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97afaeae-78b1-4b41-a808-8a1ca79729c3 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:06.835757 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97afaeae-78b1-4b41-a808-8a1ca79729c3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:06.837214 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97afaeae-78b1-4b41-a808-8a1ca79729c3 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:06.862825 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 297/595] 10.222.0.16 () {56 vars in 1137 bytes} [Tue Jul 12 17:20:06 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:06.863750 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97afaeae-78b1-4b41-a808-8a1ca79729c3 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:06.883864 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7edb2375-3702-4fae-a134-9c34c8e1c5a4 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:06.921079 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f231888-2831-48bf-b36f-630ca99f0802 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:06.922011 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f231888-2831-48bf-b36f-630ca99f0802 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:06.922374 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f231888-2831-48bf-b36f-630ca99f0802 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:06.922822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f231888-2831-48bf-b36f-630ca99f0802 None admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:06.923697 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f231888-2831-48bf-b36f-630ca99f0802 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:06.924082 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f231888-2831-48bf-b36f-630ca99f0802 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:06.925921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f231888-2831-48bf-b36f-630ca99f0802 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:06.939637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 299/596] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Jul 12 17:20:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:06.940717 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f231888-2831-48bf-b36f-630ca99f0802 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:06.977721 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97afaeae-78b1-4b41-a808-8a1ca79729c3 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:07.018590 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4147d02-adad-4749-af67-242b067cffc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:07.019394 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4147d02-adad-4749-af67-242b067cffc5 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:07.019689 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4147d02-adad-4749-af67-242b067cffc5 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:07.019972 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4147d02-adad-4749-af67-242b067cffc5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:07.031344 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4147d02-adad-4749-af67-242b067cffc5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:07.033157 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4147d02-adad-4749-af67-242b067cffc5 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:07.039926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:07.053227 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-d4147d02-adad-4749-af67-242b067cffc5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:07.053227 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:07.053227 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:07.053227 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:07.053227 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:07.053227 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:07.053227 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:07.053227 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:07.053227 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:07.053227 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:07.053227 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:07.053227 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:07.053227 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:07.053227 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:07.053227 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:07.061989 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:07.061989 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:07.061989 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:07.061989 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:07.061989 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:07.061989 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:07.061989 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:07.061989 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:07.061989 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:07.061989 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:07.061989 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:07.061989 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:07.061989 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:07.061989 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:07.061989 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:07.061989 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:07.061989 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:07.063491 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:07.063491 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:07.063491 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:07.063491 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:07.063491 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 298/597] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:07.063491 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4147d02-adad-4749-af67-242b067cffc5 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:07.066081 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f231888-2831-48bf-b36f-630ca99f0802 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:07.075899 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:07.106948 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5572b6c-61f8-41d0-9e07-cf2e150b73ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:07.108034 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5572b6c-61f8-41d0-9e07-cf2e150b73ef None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:07.108487 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5572b6c-61f8-41d0-9e07-cf2e150b73ef None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:07.108928 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5572b6c-61f8-41d0-9e07-cf2e150b73ef None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:07.120526 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5572b6c-61f8-41d0-9e07-cf2e150b73ef None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:07.123423 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5572b6c-61f8-41d0-9e07-cf2e150b73ef None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:07.143673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-d5572b6c-61f8-41d0-9e07-cf2e150b73ef None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:07.143673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:07.143673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:07.143673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:07.143673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:07.143673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:07.143673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:07.143673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:07.143673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:07.143673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:07.143673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:07.143673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:07.143673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:07.143673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:07.143673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:07.144963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:07.144963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:07.144963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:07.144963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:07.144963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:07.144963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:07.144963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:07.144963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:07.144963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:07.144963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:07.144963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:07.144963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:07.144963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:07.144963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:07.144963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:07.144963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:07.144963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:07.146063 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:07.146063 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:07.146063 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:07.146063 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:07.148146 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 300/598] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:07.149077 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5572b6c-61f8-41d0-9e07-cf2e150b73ef None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:07.174987 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4147d02-adad-4749-af67-242b067cffc5 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:07.213450 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084abf9b-4c45-47e2-a38a-006315a5ea87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:07.214461 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084abf9b-4c45-47e2-a38a-006315a5ea87 None admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:07.214775 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084abf9b-4c45-47e2-a38a-006315a5ea87 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:07.215064 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-084abf9b-4c45-47e2-a38a-006315a5ea87 None admin] PATH_INFO: `/v3/projects/fd4ed2fe037c4fb7ba49c724eb9414c2/users/4018d94a1feb4d6dade88ee9663f9729/roles/bef74d4975164f75bcec96ac47afea73` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:07.231571 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084abf9b-4c45-47e2-a38a-006315a5ea87 None admin] RBAC: Authorizing `identity:create_grant(project_id=fd4ed2fe037c4fb7ba49c724eb9414c2, user_id=4018d94a1feb4d6dade88ee9663f9729, role_id=bef74d4975164f75bcec96ac47afea73)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:07.233593 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-084abf9b-4c45-47e2-a38a-006315a5ea87 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:07.268779 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5572b6c-61f8-41d0-9e07-cf2e150b73ef None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:07.304615 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b23e9b52-c544-42ea-8e81-96f6cc68e3dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:07.305535 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23e9b52-c544-42ea-8e81-96f6cc68e3dd None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:07.306043 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23e9b52-c544-42ea-8e81-96f6cc68e3dd None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:07.306756 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23e9b52-c544-42ea-8e81-96f6cc68e3dd None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:07.307701 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b23e9b52-c544-42ea-8e81-96f6cc68e3dd None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:07.308286 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b23e9b52-c544-42ea-8e81-96f6cc68e3dd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:07.309924 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b23e9b52-c544-42ea-8e81-96f6cc68e3dd None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:07.311781 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:07.338529 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 299/599] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Jul 12 17:20:07 2022] PUT /identity/v3/projects/fd4ed2fe037c4fb7ba49c724eb9414c2/users/4018d94a1feb4d6dade88ee9663f9729/roles/bef74d4975164f75bcec96ac47afea73 => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:20:07.339065 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 301/600] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:07.339178 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084abf9b-4c45-47e2-a38a-006315a5ea87 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:07.339257 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b23e9b52-c544-42ea-8e81-96f6cc68e3dd None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:07.446747 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b23e9b52-c544-42ea-8e81-96f6cc68e3dd None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:07.455391 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:07.493577 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf2780c-f83d-4fcb-8af3-18c75e199374 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:07.494755 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf2780c-f83d-4fcb-8af3-18c75e199374 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:07.496791 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf2780c-f83d-4fcb-8af3-18c75e199374 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:07.497313 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf2780c-f83d-4fcb-8af3-18c75e199374 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:07.511138 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf2780c-f83d-4fcb-8af3-18c75e199374 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:07.514422 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bf2780c-f83d-4fcb-8af3-18c75e199374 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:07.538752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-9bf2780c-f83d-4fcb-8af3-18c75e199374 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:07.538752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:07.538752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:07.538752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:07.538752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:07.538752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:07.538752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:07.538752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:07.538752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:07.538752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:07.538752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:07.538752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:07.538752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:20:07.538752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:20:07.538752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:20:07.539726 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:20:07.539726 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:07.539726 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:07.539726 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:07.539726 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:07.539726 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:07.539726 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:07.539726 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:07.539726 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:07.539726 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:07.539726 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:07.539726 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:07.539726 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:07.539726 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:07.539726 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:07.539726 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:07.539726 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:07.540611 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:20:07.540611 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:20:07.540611 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:20:07.540611 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:20:07.540611 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:20:07.540611 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:20:07.540611 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:07.540611 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:07.541093 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-084abf9b-4c45-47e2-a38a-006315a5ea87 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:07.543311 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 302/601] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:20:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:07.549812 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf2780c-f83d-4fcb-8af3-18c75e199374 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:07.616445 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:07.632771 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01479860-2525-4690-80c7-4eca332a274f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:07.633610 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01479860-2525-4690-80c7-4eca332a274f None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:07.633965 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01479860-2525-4690-80c7-4eca332a274f None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:07.634357 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01479860-2525-4690-80c7-4eca332a274f None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:07.635246 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01479860-2525-4690-80c7-4eca332a274f None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:07.635599 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01479860-2525-4690-80c7-4eca332a274f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:07.637066 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01479860-2525-4690-80c7-4eca332a274f None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:07.651394 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 300/602] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 324 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:07.657842 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01479860-2525-4690-80c7-4eca332a274f None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:07.678971 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bf2780c-f83d-4fcb-8af3-18c75e199374 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:07.723850 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5846b110-ef9f-4b29-abef-afa078299a0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:07.724531 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5846b110-ef9f-4b29-abef-afa078299a0d None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:07.724822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5846b110-ef9f-4b29-abef-afa078299a0d None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:07.725104 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5846b110-ef9f-4b29-abef-afa078299a0d None admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:07.725911 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5846b110-ef9f-4b29-abef-afa078299a0d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:07.726231 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5846b110-ef9f-4b29-abef-afa078299a0d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:07.727709 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5846b110-ef9f-4b29-abef-afa078299a0d None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:07.759222 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 303/603] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Jul 12 17:20:07 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:20:07.765223 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5846b110-ef9f-4b29-abef-afa078299a0d None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:07.881889 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01479860-2525-4690-80c7-4eca332a274f None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:07.899468 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5846b110-ef9f-4b29-abef-afa078299a0d None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:07.920292 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:07.952200 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37897368-ace5-42da-a651-179f5ffdf1f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:07.953196 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37897368-ace5-42da-a651-179f5ffdf1f8 None admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:07.953483 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37897368-ace5-42da-a651-179f5ffdf1f8 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:07.953759 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37897368-ace5-42da-a651-179f5ffdf1f8 None admin] PATH_INFO: `/v3/projects/fd4ed2fe037c4fb7ba49c724eb9414c2/users/bf5bf02ae13d4d86a1b157369ef07a00/roles/bef74d4975164f75bcec96ac47afea73` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:07.962812 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6d01e4-d1b2-43e0-9cc6-24f6ddde0d65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:07.964471 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6d01e4-d1b2-43e0-9cc6-24f6ddde0d65 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:07.964765 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6d01e4-d1b2-43e0-9cc6-24f6ddde0d65 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:07.965051 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6d01e4-d1b2-43e0-9cc6-24f6ddde0d65 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:07.970779 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:07.981362 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f6d01e4-d1b2-43e0-9cc6-24f6ddde0d65 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:07.985396 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f6d01e4-d1b2-43e0-9cc6-24f6ddde0d65 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:07.988629 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37897368-ace5-42da-a651-179f5ffdf1f8 None admin] RBAC: Authorizing `identity:create_grant(project_id=fd4ed2fe037c4fb7ba49c724eb9414c2, user_id=bf5bf02ae13d4d86a1b157369ef07a00, role_id=bef74d4975164f75bcec96ac47afea73)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:07.990139 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37897368-ace5-42da-a651-179f5ffdf1f8 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:08.009260 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-4f6d01e4-d1b2-43e0-9cc6-24f6ddde0d65 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:08.009260 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:08.009260 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:08.009260 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:08.009260 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:08.009260 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:08.009260 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:08.009260 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:08.009260 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:08.009260 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:08.009260 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:08.009260 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:08.009260 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:20:08.009260 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:20:08.009260 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:20:08.010259 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:20:08.010259 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:08.010259 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:08.010259 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:08.010259 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:08.010259 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:08.010259 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:08.010259 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:08.010259 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:08.010259 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:08.010259 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:08.010259 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:08.010259 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:08.010259 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:08.010259 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:08.010259 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:08.010259 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:08.011230 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:20:08.011230 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:20:08.011230 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:20:08.011230 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:20:08.011230 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:20:08.011230 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:20:08.011230 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:08.011230 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:08.011230 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 301/604] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:20:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 354 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:20:08.017029 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6d01e4-d1b2-43e0-9cc6-24f6ddde0d65 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:08.137198 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6d01e4-d1b2-43e0-9cc6-24f6ddde0d65 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:08.137770 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 304/605] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Jul 12 17:20:07 2022] PUT /identity/v3/projects/fd4ed2fe037c4fb7ba49c724eb9414c2/users/bf5bf02ae13d4d86a1b157369ef07a00/roles/bef74d4975164f75bcec96ac47afea73 => generated 0 bytes in 367 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:20:08.139448 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25f99e1-5a4e-4370-a6d9-c5e720bd769c None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:08.139837 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25f99e1-5a4e-4370-a6d9-c5e720bd769c None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:08.140214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25f99e1-5a4e-4370-a6d9-c5e720bd769c None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:08.150147 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 305/606] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:08 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:08.152173 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22afabe0-d2ed-469f-84b1-c1e80aa0742d None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:08.152597 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22afabe0-d2ed-469f-84b1-c1e80aa0742d None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:08.153399 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22afabe0-d2ed-469f-84b1-c1e80aa0742d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:08.188724 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1815fa64-9c87-43b3-acd8-863b6a122eda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:08.189770 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1815fa64-9c87-43b3-acd8-863b6a122eda None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:08.190263 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1815fa64-9c87-43b3-acd8-863b6a122eda None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:08.190736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1815fa64-9c87-43b3-acd8-863b6a122eda None admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:08.191794 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1815fa64-9c87-43b3-acd8-863b6a122eda None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:08.192295 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1815fa64-9c87-43b3-acd8-863b6a122eda None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:08.194084 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1815fa64-9c87-43b3-acd8-863b6a122eda None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:08.194226 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-22afabe0-d2ed-469f-84b1-c1e80aa0742d None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:08.195647 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-22afabe0-d2ed-469f-84b1-c1e80aa0742d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:08.210037 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 302/607] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Jul 12 17:20:08 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:08.217011 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1815fa64-9c87-43b3-acd8-863b6a122eda None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:08.316327 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:08.341179 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 306/608] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:20:08.373574 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22afabe0-d2ed-469f-84b1-c1e80aa0742d None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:08.386535 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1815fa64-9c87-43b3-acd8-863b6a122eda None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:08.427369 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47209408-da0b-4af1-aa6a-afdafbeea0f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:08.428193 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47209408-da0b-4af1-aa6a-afdafbeea0f3 None admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:08.428486 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47209408-da0b-4af1-aa6a-afdafbeea0f3 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:08.428768 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47209408-da0b-4af1-aa6a-afdafbeea0f3 None admin] PATH_INFO: `/v3/projects/fd4ed2fe037c4fb7ba49c724eb9414c2/users/f6a82b49464846d1a207189f17fd4cf7/roles/bef74d4975164f75bcec96ac47afea73` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:08.446954 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47209408-da0b-4af1-aa6a-afdafbeea0f3 None admin] RBAC: Authorizing `identity:create_grant(project_id=fd4ed2fe037c4fb7ba49c724eb9414c2, user_id=f6a82b49464846d1a207189f17fd4cf7, role_id=bef74d4975164f75bcec96ac47afea73)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:08.449262 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47209408-da0b-4af1-aa6a-afdafbeea0f3 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:08.489651 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:08.496485 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22afabe0-d2ed-469f-84b1-c1e80aa0742d None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:08.531914 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b835bd5-d2bc-4592-886d-9f58c7fe3254 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:08.532715 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b835bd5-d2bc-4592-886d-9f58c7fe3254 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:08.533007 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b835bd5-d2bc-4592-886d-9f58c7fe3254 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:08.533287 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b835bd5-d2bc-4592-886d-9f58c7fe3254 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:08.544537 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b835bd5-d2bc-4592-886d-9f58c7fe3254 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:20:08.549459 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 303/609] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Jul 12 17:20:08 2022] PUT /identity/v3/projects/fd4ed2fe037c4fb7ba49c724eb9414c2/users/f6a82b49464846d1a207189f17fd4cf7/roles/bef74d4975164f75bcec96ac47afea73 => generated 0 bytes in 333 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:20:08.554141 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-6b835bd5-d2bc-4592-886d-9f58c7fe3254 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:20:08.554141 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:08.554141 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:08.554141 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:08.554141 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:08.554141 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:08.554141 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:08.554141 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:08.554141 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:08.554141 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:08.554141 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:08.554141 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:08.554141 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:20:08.554141 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:20:08.554141 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:20:08.554141 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:20:08.555138 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:20:08.555138 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:20:08.555138 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:20:08.555138 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:20:08.555138 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:08.555138 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:08.555138 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:08.555138 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:08.555138 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:08.555138 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:08.555138 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:08.555138 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:08.555138 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:08.555138 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:08.555138 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:08.555138 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:08.555979 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:08.555979 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:08.555979 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:08.555979 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:08.555979 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:20:08.555979 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:20:08.555979 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:20:08.555979 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:20:08.555979 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:20:08.555979 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:20:08.555979 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:20:08.555979 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:08.562731 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 307/610] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Jul 12 17:20:08 2022] GET /identity/v3/roles/service => generated 85 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:20:08.565701 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b835bd5-d2bc-4592-886d-9f58c7fe3254 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:08.581262 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef65702-7806-430c-a799-ae4c1cdfc126 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:08.581913 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef65702-7806-430c-a799-ae4c1cdfc126 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:08.582403 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef65702-7806-430c-a799-ae4c1cdfc126 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:08.589485 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 304/611] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:08 2022] GET /identity => generated 270 bytes in 39 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:08.595016 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe02901d-2d4f-43d8-90ae-7ea8abd6c663 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:08.595502 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe02901d-2d4f-43d8-90ae-7ea8abd6c663 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:08.595854 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe02901d-2d4f-43d8-90ae-7ea8abd6c663 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:08.633891 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-fe02901d-2d4f-43d8-90ae-7ea8abd6c663 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:08.633891 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-fe02901d-2d4f-43d8-90ae-7ea8abd6c663 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:08.706548 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b835bd5-d2bc-4592-886d-9f58c7fe3254 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:08.749457 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e82d1d-e123-4be4-bea7-0e32ffa48ba1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:08.751310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e82d1d-e123-4be4-bea7-0e32ffa48ba1 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:08.751744 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e82d1d-e123-4be4-bea7-0e32ffa48ba1 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:08.752453 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e82d1d-e123-4be4-bea7-0e32ffa48ba1 None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:08.753808 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e82d1d-e123-4be4-bea7-0e32ffa48ba1 None admin] RBAC: Adding query filter params (name=service) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:08.754271 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e82d1d-e123-4be4-bea7-0e32ffa48ba1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:08.756037 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e82d1d-e123-4be4-bea7-0e32ffa48ba1 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:08.772295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 308/612] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Jul 12 17:20:08 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:08.774111 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e82d1d-e123-4be4-bea7-0e32ffa48ba1 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:08.796186 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 305/613] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:20:08.799610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:08.832397 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe02901d-2d4f-43d8-90ae-7ea8abd6c663 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:08.874240 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:08.905522 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e82d1d-e123-4be4-bea7-0e32ffa48ba1 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:08.957055 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c5ab6f-5ea5-48c8-ac92-620929491d2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:08.958264 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c5ab6f-5ea5-48c8-ac92-620929491d2e None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:08.959084 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c5ab6f-5ea5-48c8-ac92-620929491d2e None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:08.959763 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c5ab6f-5ea5-48c8-ac92-620929491d2e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:08.964905 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe02901d-2d4f-43d8-90ae-7ea8abd6c663 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:08.973471 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5c5ab6f-5ea5-48c8-ac92-620929491d2e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:08.975287 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5c5ab6f-5ea5-48c8-ac92-620929491d2e None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:08.999205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-e5c5ab6f-5ea5-48c8-ac92-620929491d2e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:08.999205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:08.999205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:08.999205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:08.999205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:08.999205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:08.999205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:08.999205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:08.999205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:08.999205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:08.999205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:08.999205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:08.999205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:08.999205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:08.999205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:09.000634 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:09.000634 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:09.000634 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:09.000634 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:09.000634 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:09.000634 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:09.000634 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:09.000634 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:09.000634 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:09.000634 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:09.000634 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:09.000634 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:09.000634 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:09.000634 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:09.000634 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:09.000634 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:09.000634 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:09.001534 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:09.001534 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:09.001534 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:09.001534 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:09.008588 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 309/614] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:20:09.010611 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c5ab6f-5ea5-48c8-ac92-620929491d2e None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:09.032205 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b7268d-3157-45f2-935f-989c25e1db76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:09.032899 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b7268d-3157-45f2-935f-989c25e1db76 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:09.033196 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b7268d-3157-45f2-935f-989c25e1db76 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:09.033480 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b7268d-3157-45f2-935f-989c25e1db76 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:09.059442 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b7268d-3157-45f2-935f-989c25e1db76 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:20:09.073846 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-13b7268d-3157-45f2-935f-989c25e1db76 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:20:09.073846 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:09.073846 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:09.073846 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:09.073846 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:09.073846 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:09.073846 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:09.073846 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:09.073846 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:09.073846 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:09.073846 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:09.073846 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:09.073846 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:20:09.073846 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:20:09.073846 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:20:09.073846 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:20:09.074886 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:20:09.074886 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:20:09.074886 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:20:09.074886 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:20:09.074886 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:09.074886 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:09.074886 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:09.074886 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:09.074886 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:09.074886 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:09.074886 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:09.074886 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:09.074886 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:09.074886 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:09.074886 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:09.074886 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:09.076405 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:09.076405 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:09.076405 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:09.076405 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:09.076405 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:20:09.076405 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:20:09.076405 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:20:09.076405 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:20:09.076405 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:20:09.076405 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:20:09.076405 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:20:09.076405 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:09.076405 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 306/615] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Jul 12 17:20:08 2022] GET /identity/v3/roles/service => generated 85 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:09.081205 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b7268d-3157-45f2-935f-989c25e1db76 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:09.099590 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:09.134142 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c5ab6f-5ea5-48c8-ac92-620929491d2e None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:09.174000 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-161bd57f-338e-408c-b218-cf3448a1100d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:09.176013 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161bd57f-338e-408c-b218-cf3448a1100d None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:09.176623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161bd57f-338e-408c-b218-cf3448a1100d None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:09.177142 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161bd57f-338e-408c-b218-cf3448a1100d None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:09.178067 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-161bd57f-338e-408c-b218-cf3448a1100d None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:09.178710 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-161bd57f-338e-408c-b218-cf3448a1100d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:09.180305 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-161bd57f-338e-408c-b218-cf3448a1100d None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:09.199592 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 310/616] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:20:09.201622 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-161bd57f-338e-408c-b218-cf3448a1100d None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:09.316349 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-161bd57f-338e-408c-b218-cf3448a1100d None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:09.347071 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:09.357225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:09.360741 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461a137d-f391-4579-94e7-bde1999c9567 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:09.361754 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461a137d-f391-4579-94e7-bde1999c9567 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:09.368379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461a137d-f391-4579-94e7-bde1999c9567 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:09.368379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461a137d-f391-4579-94e7-bde1999c9567 None admin] PATH_INFO: `/v3/users/swift` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:09.373706 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461a137d-f391-4579-94e7-bde1999c9567 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:09.374728 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b7268d-3157-45f2-935f-989c25e1db76 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:09.375876 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461a137d-f391-4579-94e7-bde1999c9567 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:09.395998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-461a137d-f391-4579-94e7-bde1999c9567 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 12 17:20:09.395998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:09.395998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:09.395998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:09.395998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:09.395998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:09.395998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:09.395998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:09.395998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:09.395998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:09.395998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:09.395998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:09.395998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:20:09.395998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:20:09.395998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:20:09.395998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:20:09.396986 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:09.396986 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:09.396986 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:20:09.396986 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:09.396986 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:20:09.396986 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:09.396986 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:09.396986 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:09.396986 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:09.396986 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:09.396986 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:09.396986 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:09.396986 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:09.396986 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:09.396986 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:09.396986 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:09.396986 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:09.396986 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:09.397905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:09.397905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:09.397905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:20:09.397905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:20:09.397905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:20:09.397905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:20:09.397905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:20:09.397905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:20:09.397905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 12 17:20:09.397905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:09.405455 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 311/617] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Jul 12 17:20:09 2022] GET /identity/v3/users/swift => generated 83 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:09.407840 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461a137d-f391-4579-94e7-bde1999c9567 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:09.439782 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110d08cd-4574-4a1c-9ae4-cd2450087e7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:09.440817 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110d08cd-4574-4a1c-9ae4-cd2450087e7b None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:09.440936 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110d08cd-4574-4a1c-9ae4-cd2450087e7b None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:09.441133 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-110d08cd-4574-4a1c-9ae4-cd2450087e7b None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:09.441955 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110d08cd-4574-4a1c-9ae4-cd2450087e7b None admin] RBAC: Adding query filter params (name=service) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:09.442284 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110d08cd-4574-4a1c-9ae4-cd2450087e7b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:09.448155 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-110d08cd-4574-4a1c-9ae4-cd2450087e7b None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:09.466057 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 307/618] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Jul 12 17:20:09 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 385 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:09.472151 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110d08cd-4574-4a1c-9ae4-cd2450087e7b None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:09.526734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461a137d-f391-4579-94e7-bde1999c9567 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:09.563803 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe08fa10-a723-4c32-8657-62f0957e2e9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:09.564965 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe08fa10-a723-4c32-8657-62f0957e2e9f None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:09.565452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe08fa10-a723-4c32-8657-62f0957e2e9f None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:09.565928 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe08fa10-a723-4c32-8657-62f0957e2e9f None admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:09.567081 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe08fa10-a723-4c32-8657-62f0957e2e9f None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:09.567639 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe08fa10-a723-4c32-8657-62f0957e2e9f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:09.569298 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe08fa10-a723-4c32-8657-62f0957e2e9f None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:09.610703 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 312/619] 10.222.0.16 () {56 vars in 1135 bytes} [Tue Jul 12 17:20:09 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:20:09.615305 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe08fa10-a723-4c32-8657-62f0957e2e9f None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:09.666944 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:09.689353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-110d08cd-4574-4a1c-9ae4-cd2450087e7b None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:09.724040 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe08fa10-a723-4c32-8657-62f0957e2e9f None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:09.747937 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2473de8-b538-4488-aae0-87d0310e7848 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:09.749357 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2473de8-b538-4488-aae0-87d0310e7848 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:09.750493 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2473de8-b538-4488-aae0-87d0310e7848 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:09.751004 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2473de8-b538-4488-aae0-87d0310e7848 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:09.761324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9813d0f-f0b0-4811-a93e-d24b5d36d25f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:09.762215 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9813d0f-f0b0-4811-a93e-d24b5d36d25f None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:09.762758 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9813d0f-f0b0-4811-a93e-d24b5d36d25f None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:09.763131 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9813d0f-f0b0-4811-a93e-d24b5d36d25f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:09.774889 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9813d0f-f0b0-4811-a93e-d24b5d36d25f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:09.776098 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2473de8-b538-4488-aae0-87d0310e7848 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:09.776597 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9813d0f-f0b0-4811-a93e-d24b5d36d25f None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:09.777932 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2473de8-b538-4488-aae0-87d0310e7848 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:09.795563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-b9813d0f-f0b0-4811-a93e-d24b5d36d25f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:09.795563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:09.795563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:09.795563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:09.795563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:09.795563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:09.795563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:09.795563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:09.795563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:09.795563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:09.795563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:09.795563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:09.795563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:09.795563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:09.795563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:09.796921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:09.796921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:09.796921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:09.796921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:09.796921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:09.796921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:09.796921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:09.796921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:09.796921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:09.796921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:09.796921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:09.796921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:09.796921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:09.796921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:09.796921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:09.796921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:09.796921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:09.798082 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:09.798082 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:09.798082 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:09.798082 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:09.802606 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 313/620] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:09.805157 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9813d0f-f0b0-4811-a93e-d24b5d36d25f None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:09.811277 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-c2473de8-b538-4488-aae0-87d0310e7848 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:09.811277 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:09.811277 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:09.811277 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:09.811277 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:09.811277 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:09.811277 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:09.811277 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:09.811277 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:09.811277 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:09.811277 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:09.811277 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:09.811277 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:09.811277 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:09.811277 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:09.812974 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:09.812974 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:09.812974 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:09.812974 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:09.812974 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:09.812974 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:09.812974 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:09.812974 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:09.812974 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:09.812974 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:09.812974 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:09.812974 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:09.812974 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:09.812974 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:09.812974 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:09.812974 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:09.812974 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:09.814081 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:09.814081 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:09.814081 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:09.814081 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:09.822199 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 308/621] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 350 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:20:09.827420 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2473de8-b538-4488-aae0-87d0310e7848 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:09.833775 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:09.922417 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:09.926864 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9813d0f-f0b0-4811-a93e-d24b5d36d25f None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:09.970174 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaaa0209-a45b-4141-8230-68ef0eaab2db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:09.970928 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaaa0209-a45b-4141-8230-68ef0eaab2db None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:09.971223 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaaa0209-a45b-4141-8230-68ef0eaab2db None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:09.971507 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaaa0209-a45b-4141-8230-68ef0eaab2db None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:09.972390 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaaa0209-a45b-4141-8230-68ef0eaab2db None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:09.973015 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaaa0209-a45b-4141-8230-68ef0eaab2db None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:09.974738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaaa0209-a45b-4141-8230-68ef0eaab2db None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:09.975682 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2473de8-b538-4488-aae0-87d0310e7848 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:09.993337 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 314/622] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:20:09.993337 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50f0217-047f-4215-b4c8-9dd33de10d1e None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:09.993337 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50f0217-047f-4215-b4c8-9dd33de10d1e None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:09.993337 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50f0217-047f-4215-b4c8-9dd33de10d1e None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:09.993337 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 315/623] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:09.996595 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50f0217-047f-4215-b4c8-9dd33de10d1e None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:10.027858 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-927c1a5a-4a47-4131-b2cb-c27f51645d89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:10.029500 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927c1a5a-4a47-4131-b2cb-c27f51645d89 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:10.029985 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927c1a5a-4a47-4131-b2cb-c27f51645d89 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:10.030767 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927c1a5a-4a47-4131-b2cb-c27f51645d89 None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:10.032292 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-927c1a5a-4a47-4131-b2cb-c27f51645d89 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:10.033178 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-927c1a5a-4a47-4131-b2cb-c27f51645d89 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:10.035263 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-927c1a5a-4a47-4131-b2cb-c27f51645d89 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:10.057314 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 309/624] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:20:10.059029 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2268aecb-235e-446c-a769-bb9c8f21f114 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:10.059464 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2268aecb-235e-446c-a769-bb9c8f21f114 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:10.059959 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2268aecb-235e-446c-a769-bb9c8f21f114 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:10.079322 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-2268aecb-235e-446c-a769-bb9c8f21f114 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:10.080577 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-2268aecb-235e-446c-a769-bb9c8f21f114 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:10.117883 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50f0217-047f-4215-b4c8-9dd33de10d1e None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:10.146034 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:10.157425 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 310/625] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:20:10.163278 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2268aecb-235e-446c-a769-bb9c8f21f114 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:10.165060 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-977fc7e2-a027-4be9-9042-b0a661e8a3c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:10.166033 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977fc7e2-a027-4be9-9042-b0a661e8a3c6 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:10.168242 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977fc7e2-a027-4be9-9042-b0a661e8a3c6 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:10.168672 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977fc7e2-a027-4be9-9042-b0a661e8a3c6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:10.182129 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-977fc7e2-a027-4be9-9042-b0a661e8a3c6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:10.184338 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-977fc7e2-a027-4be9-9042-b0a661e8a3c6 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:10.200711 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-977fc7e2-a027-4be9-9042-b0a661e8a3c6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:10.200711 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:10.200711 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:10.200711 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:10.200711 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:10.200711 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:10.200711 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:10.200711 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:10.200711 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:10.200711 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:10.200711 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:10.200711 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:10.200711 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:20:10.200711 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:20:10.200711 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:20:10.201661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:20:10.201661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:10.201661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:10.201661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:10.201661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:10.201661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:10.201661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:10.201661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:10.201661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:10.201661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:10.201661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:10.201661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:10.201661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:10.201661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:10.201661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:10.201661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:10.201661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:10.202606 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:20:10.202606 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:20:10.202606 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:20:10.202606 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:20:10.202606 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:20:10.202606 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:20:10.202606 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:10.202606 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:10.209963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 316/626] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:20:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:20:10.211064 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-977fc7e2-a027-4be9-9042-b0a661e8a3c6 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:10.237229 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:10.288691 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2268aecb-235e-446c-a769-bb9c8f21f114 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:10.327769 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce42c21-9caf-47fb-b1de-73aaa5ed12d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:10.329397 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce42c21-9caf-47fb-b1de-73aaa5ed12d6 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:10.329862 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce42c21-9caf-47fb-b1de-73aaa5ed12d6 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:10.330315 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce42c21-9caf-47fb-b1de-73aaa5ed12d6 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:10.338310 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce42c21-9caf-47fb-b1de-73aaa5ed12d6 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:10.340211 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce42c21-9caf-47fb-b1de-73aaa5ed12d6 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:10.347310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-977fc7e2-a027-4be9-9042-b0a661e8a3c6 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:10.356632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-dce42c21-9caf-47fb-b1de-73aaa5ed12d6 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 12 17:20:10.356632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:10.356632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:10.356632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:10.356632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:10.356632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:10.356632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:10.356632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:10.356632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:10.356632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:10.356632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:10.356632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:10.356632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:20:10.356632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:20:10.356632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:20:10.356632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:20:10.357585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:10.357585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:10.357585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:20:10.357585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:10.357585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:20:10.357585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:10.357585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:10.357585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:10.357585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:10.357585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:10.357585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:10.357585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:10.357585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:10.357585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:10.357585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:10.357585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:10.357585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:10.357585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:10.359572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:10.359572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:10.359572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:20:10.359572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:20:10.359572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:20:10.359572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:20:10.359572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:20:10.359572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:20:10.359572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 12 17:20:10.359572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:10.362340 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 311/627] 10.222.0.16 () {56 vars in 1093 bytes} [Tue Jul 12 17:20:10 2022] GET /identity/v3/users/nova => generated 82 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:10.363594 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce42c21-9caf-47fb-b1de-73aaa5ed12d6 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:10.396073 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7765d82d-498f-49a5-8a57-016ab720c8d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:10.396884 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7765d82d-498f-49a5-8a57-016ab720c8d1 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:10.397172 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7765d82d-498f-49a5-8a57-016ab720c8d1 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:10.397707 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7765d82d-498f-49a5-8a57-016ab720c8d1 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:10.411642 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7765d82d-498f-49a5-8a57-016ab720c8d1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:20:10.420449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-7765d82d-498f-49a5-8a57-016ab720c8d1 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:20:10.420449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:10.420449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:10.420449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:10.420449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:10.420449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:10.420449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:10.420449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:10.420449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:10.420449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:10.420449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:10.420449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:10.420449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:20:10.420449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:20:10.420449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:20:10.420449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:20:10.421904 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:20:10.421904 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:20:10.421904 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:20:10.421904 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:20:10.421904 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:10.421904 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:10.421904 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:10.421904 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:10.421904 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:10.421904 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:10.421904 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:10.421904 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:10.421904 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:10.421904 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:10.421904 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:10.421904 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:10.423186 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:10.423186 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:10.423186 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:10.423186 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:10.423186 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:20:10.423186 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:20:10.423186 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:20:10.423186 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:20:10.423186 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:20:10.423186 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:20:10.423186 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:20:10.423186 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:10.425888 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 317/628] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Jul 12 17:20:10 2022] GET /identity/v3/roles/service => generated 85 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:10.426897 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7765d82d-498f-49a5-8a57-016ab720c8d1 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:10.436600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:10.503660 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce42c21-9caf-47fb-b1de-73aaa5ed12d6 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:10.525516 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:10.543222 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1a9327-ff8c-4e1f-abf8-9d01b6d4e07d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:10.544209 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1a9327-ff8c-4e1f-abf8-9d01b6d4e07d None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:10.544695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1a9327-ff8c-4e1f-abf8-9d01b6d4e07d None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:10.545198 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc1a9327-ff8c-4e1f-abf8-9d01b6d4e07d None admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:10.546210 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc1a9327-ff8c-4e1f-abf8-9d01b6d4e07d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:10.547507 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc1a9327-ff8c-4e1f-abf8-9d01b6d4e07d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:10.549495 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc1a9327-ff8c-4e1f-abf8-9d01b6d4e07d None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:10.558977 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7765d82d-498f-49a5-8a57-016ab720c8d1 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:10.561894 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 312/629] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Jul 12 17:20:10 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:10.562688 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1a9327-ff8c-4e1f-abf8-9d01b6d4e07d None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:10.603251 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adbef62c-d4c5-4d80-8eae-556817b84b8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:10.604106 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adbef62c-d4c5-4d80-8eae-556817b84b8e None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:10.604394 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adbef62c-d4c5-4d80-8eae-556817b84b8e None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:10.604675 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adbef62c-d4c5-4d80-8eae-556817b84b8e None admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:10.605512 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adbef62c-d4c5-4d80-8eae-556817b84b8e None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:10.605827 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adbef62c-d4c5-4d80-8eae-556817b84b8e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:10.607350 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adbef62c-d4c5-4d80-8eae-556817b84b8e None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:10.630685 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 318/630] 10.222.0.16 () {56 vars in 1133 bytes} [Tue Jul 12 17:20:10 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:10.630685 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969772c0-99a7-4270-b2b9-0ea8c2c204a5 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:10.630685 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969772c0-99a7-4270-b2b9-0ea8c2c204a5 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:10.630685 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969772c0-99a7-4270-b2b9-0ea8c2c204a5 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:10.630685 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 319/631] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:10.631290 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969772c0-99a7-4270-b2b9-0ea8c2c204a5 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:10.687022 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc1a9327-ff8c-4e1f-abf8-9d01b6d4e07d None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:10.703213 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:10.727869 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4cab352-b462-4d31-8f5d-c00ccaf4f220 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:10.728794 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cab352-b462-4d31-8f5d-c00ccaf4f220 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:10.729079 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cab352-b462-4d31-8f5d-c00ccaf4f220 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:10.729357 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cab352-b462-4d31-8f5d-c00ccaf4f220 None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:10.730106 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4cab352-b462-4d31-8f5d-c00ccaf4f220 None admin] RBAC: Adding query filter params (name=service) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:10.730408 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4cab352-b462-4d31-8f5d-c00ccaf4f220 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:10.731801 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4cab352-b462-4d31-8f5d-c00ccaf4f220 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:10.745271 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969772c0-99a7-4270-b2b9-0ea8c2c204a5 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:10.745485 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 313/632] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Jul 12 17:20:10 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:10.745485 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4cab352-b462-4d31-8f5d-c00ccaf4f220 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:10.777886 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67f2f3f-f8a5-4365-9e78-f0c71bd17797 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:10.778897 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67f2f3f-f8a5-4365-9e78-f0c71bd17797 None admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:10.779182 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67f2f3f-f8a5-4365-9e78-f0c71bd17797 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:10.779459 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67f2f3f-f8a5-4365-9e78-f0c71bd17797 None admin] PATH_INFO: `/v3/projects/fd4ed2fe037c4fb7ba49c724eb9414c2/users/d7906d1a19c642bd93507a27bf3e3dae/roles/bef74d4975164f75bcec96ac47afea73` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:10.783764 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:10.796114 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67f2f3f-f8a5-4365-9e78-f0c71bd17797 None admin] RBAC: Authorizing `identity:create_grant(project_id=fd4ed2fe037c4fb7ba49c724eb9414c2, user_id=d7906d1a19c642bd93507a27bf3e3dae, role_id=bef74d4975164f75bcec96ac47afea73)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:10.798444 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67f2f3f-f8a5-4365-9e78-f0c71bd17797 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:10.852267 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4cab352-b462-4d31-8f5d-c00ccaf4f220 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:10.887132 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ec9647-8fa8-4eb5-b9fd-e5ffaeeb3d72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:10.887132 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ec9647-8fa8-4eb5-b9fd-e5ffaeeb3d72 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:10.887132 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ec9647-8fa8-4eb5-b9fd-e5ffaeeb3d72 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:10.887132 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ec9647-8fa8-4eb5-b9fd-e5ffaeeb3d72 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:10.922582 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 320/633] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Jul 12 17:20:10 2022] PUT /identity/v3/projects/fd4ed2fe037c4fb7ba49c724eb9414c2/users/d7906d1a19c642bd93507a27bf3e3dae/roles/bef74d4975164f75bcec96ac47afea73 => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:20:10.922582 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884890ec-5b78-4e5d-8a24-cd06207a5b79 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:10.922582 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884890ec-5b78-4e5d-8a24-cd06207a5b79 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:10.922582 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884890ec-5b78-4e5d-8a24-cd06207a5b79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:10.942115 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:10.944711 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-884890ec-5b78-4e5d-8a24-cd06207a5b79 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:10.946607 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-884890ec-5b78-4e5d-8a24-cd06207a5b79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:10.976443 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40ec9647-8fa8-4eb5-b9fd-e5ffaeeb3d72 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:10.978923 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40ec9647-8fa8-4eb5-b9fd-e5ffaeeb3d72 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:11.044401 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 321/634] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:20:11.045350 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884890ec-5b78-4e5d-8a24-cd06207a5b79 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:11.130779 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 314/635] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:20:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 389 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 12 17:20:11.131699 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ec9647-8fa8-4eb5-b9fd-e5ffaeeb3d72 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:11.170787 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884890ec-5b78-4e5d-8a24-cd06207a5b79 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:11.189207 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:11.208970 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077e87f9-2753-483c-9c6b-ef3e1366ab42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:11.209776 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077e87f9-2753-483c-9c6b-ef3e1366ab42 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:11.210063 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077e87f9-2753-483c-9c6b-ef3e1366ab42 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:11.210342 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077e87f9-2753-483c-9c6b-ef3e1366ab42 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:11.218836 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:11.222361 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077e87f9-2753-483c-9c6b-ef3e1366ab42 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:11.224291 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077e87f9-2753-483c-9c6b-ef3e1366ab42 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:11.242542 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-077e87f9-2753-483c-9c6b-ef3e1366ab42 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:11.242542 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:11.242542 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:11.242542 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:11.242542 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:11.242542 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:11.242542 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:11.242542 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:11.242542 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:11.242542 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:11.242542 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:11.242542 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:11.242542 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:11.242542 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:11.242542 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:11.243540 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:11.243540 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:11.243540 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:11.243540 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:11.243540 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:11.243540 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:11.243540 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:11.243540 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:11.243540 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:11.243540 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:11.243540 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:11.243540 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:11.243540 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:11.243540 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:11.243540 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:11.243540 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:11.243540 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:11.244452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:11.244452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:11.244452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:11.244452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:11.244452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 322/636] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:11.245218 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dc01fe-ba34-4367-9021-3d9f6c97ec06 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:11.245394 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dc01fe-ba34-4367-9021-3d9f6c97ec06 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:11.245673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0dc01fe-ba34-4367-9021-3d9f6c97ec06 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:11.250473 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 323/637] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:11.250473 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0dc01fe-ba34-4367-9021-3d9f6c97ec06 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:11.252367 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ec9647-8fa8-4eb5-b9fd-e5ffaeeb3d72 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:11.289654 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d83b50a-628f-43cb-82bf-bab805b2a6d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:11.290924 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d83b50a-628f-43cb-82bf-bab805b2a6d3 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:11.291212 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d83b50a-628f-43cb-82bf-bab805b2a6d3 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:11.291492 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d83b50a-628f-43cb-82bf-bab805b2a6d3 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:11.305105 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d83b50a-628f-43cb-82bf-bab805b2a6d3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:20:11.313084 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-2d83b50a-628f-43cb-82bf-bab805b2a6d3 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:20:11.313084 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:11.313084 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:11.313084 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:11.313084 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:11.313084 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:11.313084 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:11.313084 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:11.313084 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:11.313084 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:11.313084 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:11.313084 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:11.313084 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:20:11.313084 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:20:11.313084 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:20:11.313084 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:20:11.314265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:20:11.314265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:20:11.314265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:20:11.314265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:20:11.314265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:11.314265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:11.314265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:11.314265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:11.314265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:11.314265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:11.314265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:11.314265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:11.314265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:11.314265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:11.314265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:11.314265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:11.320086 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:11.320086 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:11.320086 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:11.320086 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:11.320086 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:20:11.320086 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:20:11.320086 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:20:11.320086 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:20:11.320086 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:20:11.320086 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:20:11.320086 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:20:11.320086 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:11.320086 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 315/638] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Jul 12 17:20:11 2022] GET /identity/v3/roles/service => generated 85 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:11.320086 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d83b50a-628f-43cb-82bf-bab805b2a6d3 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:11.371001 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0dc01fe-ba34-4367-9021-3d9f6c97ec06 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:11.408350 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbacd94-a86d-4b47-b609-aac396dbf514 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:11.409149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbacd94-a86d-4b47-b609-aac396dbf514 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:11.409436 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbacd94-a86d-4b47-b609-aac396dbf514 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:11.409714 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbacd94-a86d-4b47-b609-aac396dbf514 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:11.420139 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abbacd94-a86d-4b47-b609-aac396dbf514 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:11.421696 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abbacd94-a86d-4b47-b609-aac396dbf514 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:11.431859 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d83b50a-628f-43cb-82bf-bab805b2a6d3 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:11.440513 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-abbacd94-a86d-4b47-b609-aac396dbf514 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:11.440513 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:11.440513 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:11.440513 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:11.440513 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:11.440513 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:11.440513 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:11.440513 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:11.440513 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:11.440513 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:11.440513 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:11.440513 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:11.440513 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:11.440513 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:11.440513 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:11.441613 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:11.441613 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:11.441613 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:11.441613 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:11.441613 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:11.441613 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:11.441613 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:11.441613 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:11.441613 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:11.441613 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:11.441613 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:11.441613 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:11.441613 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:11.441613 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:11.441613 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:11.441613 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:11.441613 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:11.443124 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:11.443124 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:11.443124 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:11.443124 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:11.443124 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 324/639] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:11.443399 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a13fcb-7d70-4d58-9b97-b78cb0a986fb None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:11.443630 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a13fcb-7d70-4d58-9b97-b78cb0a986fb None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:11.443912 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a13fcb-7d70-4d58-9b97-b78cb0a986fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:11.460971 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-d4a13fcb-7d70-4d58-9b97-b78cb0a986fb None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:11.462161 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-d4a13fcb-7d70-4d58-9b97-b78cb0a986fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:11.473957 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175d039b-7d65-43ec-bf65-632f51014ef9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:11.474659 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175d039b-7d65-43ec-bf65-632f51014ef9 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:11.475225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175d039b-7d65-43ec-bf65-632f51014ef9 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:11.475225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175d039b-7d65-43ec-bf65-632f51014ef9 None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:11.476065 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-175d039b-7d65-43ec-bf65-632f51014ef9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:11.476384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-175d039b-7d65-43ec-bf65-632f51014ef9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:11.477033 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:11.477764 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-175d039b-7d65-43ec-bf65-632f51014ef9 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:11.479615 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:11.491432 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 316/640] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:11.492730 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175d039b-7d65-43ec-bf65-632f51014ef9 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:11.539088 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 325/641] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:20:11.540379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a13fcb-7d70-4d58-9b97-b78cb0a986fb None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:11.630321 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175d039b-7d65-43ec-bf65-632f51014ef9 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:11.663940 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a13fcb-7d70-4d58-9b97-b78cb0a986fb None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:11.670547 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70e4618-fa51-41a4-a272-66ec238f7bfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:11.671812 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70e4618-fa51-41a4-a272-66ec238f7bfb None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:11.672104 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70e4618-fa51-41a4-a272-66ec238f7bfb None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:11.672392 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70e4618-fa51-41a4-a272-66ec238f7bfb None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:11.673275 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c70e4618-fa51-41a4-a272-66ec238f7bfb None admin] RBAC: Adding query filter params (name=service) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:11.673593 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c70e4618-fa51-41a4-a272-66ec238f7bfb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:11.675000 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c70e4618-fa51-41a4-a272-66ec238f7bfb None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:11.688990 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 317/642] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Jul 12 17:20:11 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:11.688990 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70e4618-fa51-41a4-a272-66ec238f7bfb None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:11.707123 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394d5f31-db65-4e13-9699-13a65dd16a8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:11.707907 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394d5f31-db65-4e13-9699-13a65dd16a8a None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:11.708239 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394d5f31-db65-4e13-9699-13a65dd16a8a None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:11.708594 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394d5f31-db65-4e13-9699-13a65dd16a8a None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:11.709396 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-394d5f31-db65-4e13-9699-13a65dd16a8a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:11.709809 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-394d5f31-db65-4e13-9699-13a65dd16a8a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:11.711234 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-394d5f31-db65-4e13-9699-13a65dd16a8a None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:11.722921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 326/643] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:11.723514 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394d5f31-db65-4e13-9699-13a65dd16a8a None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:11.736843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:11.751871 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:11.795113 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70e4618-fa51-41a4-a272-66ec238f7bfb None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:11.829539 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740ba916-3bcc-4712-9926-08fc2bc4736a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:11.830335 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740ba916-3bcc-4712-9926-08fc2bc4736a None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:11.830566 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394d5f31-db65-4e13-9699-13a65dd16a8a None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:11.830667 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740ba916-3bcc-4712-9926-08fc2bc4736a None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:11.830933 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740ba916-3bcc-4712-9926-08fc2bc4736a None admin] PATH_INFO: `/v3/users/glance` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:11.840314 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740ba916-3bcc-4712-9926-08fc2bc4736a None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:11.841771 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740ba916-3bcc-4712-9926-08fc2bc4736a None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:11.856992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-740ba916-3bcc-4712-9926-08fc2bc4736a None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Jul 12 17:20:11.856992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:11.856992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:11.856992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:11.856992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:11.856992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:11.856992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:11.856992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:11.856992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:11.856992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:11.856992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:11.856992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:11.856992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:20:11.856992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:20:11.856992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:20:11.856992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:20:11.858010 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:11.858010 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:11.858010 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:20:11.858010 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:11.858010 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:20:11.858010 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:11.858010 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:11.858010 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:11.858010 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:11.858010 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:11.858010 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:11.858010 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:11.858010 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:11.858010 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:11.858010 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:11.858010 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:11.858010 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:11.858010 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:11.860195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:11.860195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:11.860195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:20:11.860195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:20:11.860195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:20:11.860195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:20:11.860195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:20:11.860195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:20:11.860195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Jul 12 17:20:11.860195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:11.860195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 318/644] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:20:11 2022] GET /identity/v3/users/glance => generated 84 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:11.860195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740ba916-3bcc-4712-9926-08fc2bc4736a None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:11.873197 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41e48d4-5eb2-4700-b381-c9b91fa74aa7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:11.873997 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41e48d4-5eb2-4700-b381-c9b91fa74aa7 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:11.874995 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41e48d4-5eb2-4700-b381-c9b91fa74aa7 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:11.875558 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41e48d4-5eb2-4700-b381-c9b91fa74aa7 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:11.886255 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41e48d4-5eb2-4700-b381-c9b91fa74aa7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:20:11.894172 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-f41e48d4-5eb2-4700-b381-c9b91fa74aa7 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:20:11.894172 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:11.894172 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:11.894172 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:11.894172 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:11.894172 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:11.894172 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:11.894172 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:11.894172 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:11.894172 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:11.894172 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:11.894172 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:11.894172 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:20:11.894172 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:20:11.894172 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:20:11.894172 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:20:11.895549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:20:11.895549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:20:11.895549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:20:11.895549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:20:11.895549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:11.895549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:11.895549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:11.895549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:11.895549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:11.895549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:11.895549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:11.895549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:11.895549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:11.895549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:11.895549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:11.895549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:11.896783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:11.896783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:11.896783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:11.896783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:11.896783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:20:11.896783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:20:11.896783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:20:11.896783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:20:11.896783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:20:11.896783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:20:11.896783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:20:11.896783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:11.896783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 327/645] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Jul 12 17:20:11 2022] GET /identity/v3/roles/service => generated 85 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:11.896783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41e48d4-5eb2-4700-b381-c9b91fa74aa7 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:11.969857 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:11.975982 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740ba916-3bcc-4712-9926-08fc2bc4736a None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:11.995888 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:12.006228 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41e48d4-5eb2-4700-b381-c9b91fa74aa7 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:12.018877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f08861e-7d44-4c32-a3f3-7c87db172f6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:12.019850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f08861e-7d44-4c32-a3f3-7c87db172f6e None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:12.020149 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f08861e-7d44-4c32-a3f3-7c87db172f6e None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:12.020433 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f08861e-7d44-4c32-a3f3-7c87db172f6e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:12.037441 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f08861e-7d44-4c32-a3f3-7c87db172f6e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:12.039074 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f08861e-7d44-4c32-a3f3-7c87db172f6e None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:12.044192 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e149d24-a088-45b3-b0b0-08eb3c34c623 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:12.045141 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e149d24-a088-45b3-b0b0-08eb3c34c623 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:12.045504 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e149d24-a088-45b3-b0b0-08eb3c34c623 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:12.045850 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e149d24-a088-45b3-b0b0-08eb3c34c623 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:12.053816 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e149d24-a088-45b3-b0b0-08eb3c34c623 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:12.055400 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e149d24-a088-45b3-b0b0-08eb3c34c623 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:12.060300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-8f08861e-7d44-4c32-a3f3-7c87db172f6e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:12.060300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:12.060300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:12.060300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:12.060300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:12.060300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:12.060300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:12.060300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:12.060300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:12.060300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:12.060300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:12.060300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:12.060300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:12.060300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:12.060300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:12.061519 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:12.061519 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:12.061519 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:12.061519 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:12.061519 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:12.061519 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:12.061519 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:12.061519 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:12.061519 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:12.061519 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:12.061519 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:12.061519 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:12.061519 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:12.061519 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:12.061519 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:12.061519 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:12.061519 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:12.067600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:12.067600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:12.067600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:12.067600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:12.067600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 319/646] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:12.067600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f08861e-7d44-4c32-a3f3-7c87db172f6e None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:12.071822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-0e149d24-a088-45b3-b0b0-08eb3c34c623 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:12.071822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:12.071822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:12.071822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:12.071822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:12.071822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:12.071822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:12.071822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:12.071822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:12.071822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:12.071822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:12.071822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:12.071822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:20:12.071822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:20:12.071822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:20:12.077857 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:20:12.077857 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:12.077857 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:12.077857 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:12.077857 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:12.077857 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:12.077857 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:12.077857 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:12.077857 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:12.077857 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:12.077857 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:12.077857 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:12.077857 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:12.077857 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:12.077857 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:12.077857 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:12.077857 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:12.080444 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:20:12.080444 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:20:12.080444 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:20:12.080444 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:20:12.080444 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:20:12.080444 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:20:12.080444 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:12.080444 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:12.080444 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 328/647] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:20:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:12.080444 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e149d24-a088-45b3-b0b0-08eb3c34c623 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:12.180430 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e149d24-a088-45b3-b0b0-08eb3c34c623 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:12.201216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f08861e-7d44-4c32-a3f3-7c87db172f6e None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:12.210689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:12.216475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea48e39-f742-44bc-8261-19512b01c963 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:12.217151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea48e39-f742-44bc-8261-19512b01c963 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:12.217443 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea48e39-f742-44bc-8261-19512b01c963 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:12.217726 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea48e39-f742-44bc-8261-19512b01c963 None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:12.222630 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea48e39-f742-44bc-8261-19512b01c963 None admin] RBAC: Adding query filter params (name=service) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:12.222630 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea48e39-f742-44bc-8261-19512b01c963 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:12.222630 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea48e39-f742-44bc-8261-19512b01c963 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:12.229232 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 329/648] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Jul 12 17:20:12 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:12.230026 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea48e39-f742-44bc-8261-19512b01c963 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:12.244781 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec92b45c-9986-47e9-9cbd-a5e6d262b360 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:12.245847 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec92b45c-9986-47e9-9cbd-a5e6d262b360 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:12.246290 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec92b45c-9986-47e9-9cbd-a5e6d262b360 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:12.247137 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec92b45c-9986-47e9-9cbd-a5e6d262b360 None admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:12.248170 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec92b45c-9986-47e9-9cbd-a5e6d262b360 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:12.248653 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec92b45c-9986-47e9-9cbd-a5e6d262b360 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:12.250301 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec92b45c-9986-47e9-9cbd-a5e6d262b360 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:12.268920 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 320/649] 10.222.0.16 () {56 vars in 1137 bytes} [Tue Jul 12 17:20:12 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:12.269995 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec92b45c-9986-47e9-9cbd-a5e6d262b360 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:12.299360 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:12.336680 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea48e39-f742-44bc-8261-19512b01c963 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:12.372351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07a4b01-e2aa-4424-8b63-e9980b2f14eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:12.373031 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07a4b01-e2aa-4424-8b63-e9980b2f14eb None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:12.373368 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07a4b01-e2aa-4424-8b63-e9980b2f14eb None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:12.373649 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07a4b01-e2aa-4424-8b63-e9980b2f14eb None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:12.374501 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07a4b01-e2aa-4424-8b63-e9980b2f14eb None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:12.374931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07a4b01-e2aa-4424-8b63-e9980b2f14eb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:12.376137 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec92b45c-9986-47e9-9cbd-a5e6d262b360 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:12.376413 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07a4b01-e2aa-4424-8b63-e9980b2f14eb None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:12.392578 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 330/650] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:12.393651 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07a4b01-e2aa-4424-8b63-e9980b2f14eb None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:12.427273 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba33fc2-8ae6-406a-9194-1b7dd1c50685 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:12.427981 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba33fc2-8ae6-406a-9194-1b7dd1c50685 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:12.428275 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba33fc2-8ae6-406a-9194-1b7dd1c50685 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:12.428554 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba33fc2-8ae6-406a-9194-1b7dd1c50685 None admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:12.429454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba33fc2-8ae6-406a-9194-1b7dd1c50685 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:12.429771 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba33fc2-8ae6-406a-9194-1b7dd1c50685 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:12.431438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba33fc2-8ae6-406a-9194-1b7dd1c50685 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:12.449245 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 321/651] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Jul 12 17:20:12 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:12.449245 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba33fc2-8ae6-406a-9194-1b7dd1c50685 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:12.455138 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:12.509356 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07a4b01-e2aa-4424-8b63-e9980b2f14eb None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:12.544427 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:12.549761 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75de93ba-41d3-4a6c-a136-f5262b81225d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:12.550596 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75de93ba-41d3-4a6c-a136-f5262b81225d None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:12.550887 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75de93ba-41d3-4a6c-a136-f5262b81225d None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:12.551208 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75de93ba-41d3-4a6c-a136-f5262b81225d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:12.559653 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba33fc2-8ae6-406a-9194-1b7dd1c50685 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:12.562385 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75de93ba-41d3-4a6c-a136-f5262b81225d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:12.566058 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75de93ba-41d3-4a6c-a136-f5262b81225d None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:12.589400 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-75de93ba-41d3-4a6c-a136-f5262b81225d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:12.589400 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:12.589400 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:12.589400 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:12.589400 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:12.589400 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:12.589400 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:12.589400 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:12.589400 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:12.589400 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:12.589400 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:12.589400 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:12.589400 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:12.589400 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:12.589400 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:12.590546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:12.590546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:12.590546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:12.590546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:12.590546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:12.590546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:12.590546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:12.590546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:12.590546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:12.590546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:12.590546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:12.590546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:12.590546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:12.590546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:12.590546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:12.590546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:12.590546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:12.596651 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:12.596651 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:12.596651 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:12.596651 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:12.596651 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 331/652] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:12.596651 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75de93ba-41d3-4a6c-a136-f5262b81225d None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:12.605790 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16327fd2-4e37-494d-9aa7-adb591842d5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:12.606610 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16327fd2-4e37-494d-9aa7-adb591842d5c None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:12.607009 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16327fd2-4e37-494d-9aa7-adb591842d5c None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:12.607407 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16327fd2-4e37-494d-9aa7-adb591842d5c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:12.684873 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16327fd2-4e37-494d-9aa7-adb591842d5c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:12.686399 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16327fd2-4e37-494d-9aa7-adb591842d5c None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:12.706075 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75de93ba-41d3-4a6c-a136-f5262b81225d None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:12.709326 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:12.742525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2d15a2-ae31-45cd-81cf-9cbae47b8dbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:12.743430 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2d15a2-ae31-45cd-81cf-9cbae47b8dbd None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:12.743849 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2d15a2-ae31-45cd-81cf-9cbae47b8dbd None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:12.744609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f2d15a2-ae31-45cd-81cf-9cbae47b8dbd None admin] PATH_INFO: `/v3/users/neutron` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:12.753735 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f2d15a2-ae31-45cd-81cf-9cbae47b8dbd None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:12.755235 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f2d15a2-ae31-45cd-81cf-9cbae47b8dbd None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:12.770509 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-1f2d15a2-ae31-45cd-81cf-9cbae47b8dbd None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Jul 12 17:20:12.770509 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:12.770509 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:12.770509 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:12.770509 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:12.770509 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:12.770509 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:12.770509 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:12.770509 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:12.770509 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:12.770509 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:12.770509 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:12.770509 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:20:12.770509 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:20:12.770509 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:20:12.770509 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:20:12.771493 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:12.771493 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:12.771493 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:20:12.771493 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:12.771493 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:20:12.771493 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:12.771493 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:12.771493 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:12.771493 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:12.771493 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:12.771493 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:12.771493 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:12.771493 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:12.771493 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:12.771493 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:12.771493 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:12.771493 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:12.771493 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:12.772489 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:12.772489 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:12.772489 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:20:12.772489 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:20:12.772489 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:20:12.772489 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:20:12.772489 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:20:12.772489 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:20:12.772489 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Jul 12 17:20:12.772489 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:12.772489 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 332/653] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Jul 12 17:20:12 2022] GET /identity/v3/users/neutron => generated 85 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:12.772489 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2d15a2-ae31-45cd-81cf-9cbae47b8dbd None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:12.783600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:12.805219 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 322/654] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:20:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 355 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 12 17:20:12.805219 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16327fd2-4e37-494d-9aa7-adb591842d5c None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:12.879233 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f2d15a2-ae31-45cd-81cf-9cbae47b8dbd None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:12.913760 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16327fd2-4e37-494d-9aa7-adb591842d5c None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:12.915559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cab353e-be4c-4204-a9b7-bd96b981233a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:12.916274 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cab353e-be4c-4204-a9b7-bd96b981233a None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:12.916563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cab353e-be4c-4204-a9b7-bd96b981233a None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:12.916846 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cab353e-be4c-4204-a9b7-bd96b981233a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:12.917713 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cab353e-be4c-4204-a9b7-bd96b981233a None admin] RBAC: Adding query filter params (role.id=bef74d4975164f75bcec96ac47afea73, scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2, user.id=9012d5b88f554dc596582a7693c2a72a) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:12.918029 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cab353e-be4c-4204-a9b7-bd96b981233a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:12.919486 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cab353e-be4c-4204-a9b7-bd96b981233a None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:12.934468 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 333/655] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:20:12 2022] GET /identity/v3/role_assignments?user.id=9012d5b88f554dc596582a7693c2a72a&scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2&role.id=bef74d4975164f75bcec96ac47afea73 => generated 608 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:12.934468 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cab353e-be4c-4204-a9b7-bd96b981233a None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:12.967451 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ddf449-70c0-485b-9574-55bb31d200c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:12.969241 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ddf449-70c0-485b-9574-55bb31d200c3 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:12.969836 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ddf449-70c0-485b-9574-55bb31d200c3 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:12.970126 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ddf449-70c0-485b-9574-55bb31d200c3 None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:12.971001 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:12.972373 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ddf449-70c0-485b-9574-55bb31d200c3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:12.972817 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ddf449-70c0-485b-9574-55bb31d200c3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:12.974526 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ddf449-70c0-485b-9574-55bb31d200c3 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:12.988278 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 323/656] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:12.989297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ddf449-70c0-485b-9574-55bb31d200c3 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:13.043862 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:13.064652 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cab353e-be4c-4204-a9b7-bd96b981233a None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:13.107293 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ddf449-70c0-485b-9574-55bb31d200c3 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:13.111356 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70699d60-c702-4b31-94d3-3882350700e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:13.112048 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70699d60-c702-4b31-94d3-3882350700e8 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:13.112339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70699d60-c702-4b31-94d3-3882350700e8 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:13.112619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70699d60-c702-4b31-94d3-3882350700e8 None admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:13.113481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70699d60-c702-4b31-94d3-3882350700e8 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:13.113782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70699d60-c702-4b31-94d3-3882350700e8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:13.115733 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70699d60-c702-4b31-94d3-3882350700e8 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:13.134987 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 334/657] 10.222.0.16 () {56 vars in 1139 bytes} [Tue Jul 12 17:20:12 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:13.135884 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da88b046-013b-4bd4-ac8f-ae23a83b2a4f None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:13.136231 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da88b046-013b-4bd4-ac8f-ae23a83b2a4f None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:13.136452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da88b046-013b-4bd4-ac8f-ae23a83b2a4f None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:13.137700 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 335/658] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:13.138383 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da88b046-013b-4bd4-ac8f-ae23a83b2a4f None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:13.149430 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b7360a-ec1f-4446-95c6-733da35fc3ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:13.150747 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b7360a-ec1f-4446-95c6-733da35fc3ac None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:13.151070 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b7360a-ec1f-4446-95c6-733da35fc3ac None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:13.151348 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b7360a-ec1f-4446-95c6-733da35fc3ac None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:13.166890 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b7360a-ec1f-4446-95c6-733da35fc3ac None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:13.168826 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b7360a-ec1f-4446-95c6-733da35fc3ac None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:13.187711 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-97b7360a-ec1f-4446-95c6-733da35fc3ac None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:13.187711 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:13.187711 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:13.187711 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:13.187711 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:13.187711 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:13.187711 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:13.187711 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:13.187711 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:13.187711 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:13.187711 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:13.187711 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:13.187711 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:13.187711 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:13.187711 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:13.188674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:13.188674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:13.188674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:13.188674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:13.188674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:13.188674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:13.188674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:13.188674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:13.188674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:13.188674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:13.188674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:13.188674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:13.188674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:13.188674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:13.188674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:13.188674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:13.188674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:13.189571 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:13.189571 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:13.189571 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:13.189571 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:13.189571 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 324/659] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:13.190728 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b7360a-ec1f-4446-95c6-733da35fc3ac None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:13.237785 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:13.256050 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da88b046-013b-4bd4-ac8f-ae23a83b2a4f None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:13.294119 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a343381-a9c9-4607-aff4-87f85989a303 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:13.294982 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a343381-a9c9-4607-aff4-87f85989a303 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:13.295275 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a343381-a9c9-4607-aff4-87f85989a303 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:13.295560 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a343381-a9c9-4607-aff4-87f85989a303 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:13.300875 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:13.303679 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a343381-a9c9-4607-aff4-87f85989a303 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:13.304856 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b7360a-ec1f-4446-95c6-733da35fc3ac None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:13.305158 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a343381-a9c9-4607-aff4-87f85989a303 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:13.322112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-2a343381-a9c9-4607-aff4-87f85989a303 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jul 12 17:20:13.322112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:13.322112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:13.322112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:13.322112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:13.322112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:13.322112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:13.322112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:13.322112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:13.322112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:13.322112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:13.322112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:13.322112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:20:13.322112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:20:13.322112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:20:13.322112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:20:13.323332 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:13.323332 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:13.323332 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:20:13.323332 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:13.323332 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:20:13.323332 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:13.323332 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:13.323332 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:13.323332 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:13.323332 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:13.323332 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:13.323332 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:13.323332 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:13.323332 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:13.323332 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:13.323332 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:13.323332 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:13.323332 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:13.324270 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:13.324270 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:13.324270 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:20:13.324270 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:20:13.324270 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:20:13.324270 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:20:13.324270 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:20:13.324270 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:20:13.324270 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jul 12 17:20:13.324270 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:13.326859 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 336/660] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:20:13 2022] GET /identity/v3/users/cinder => generated 84 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:13.329174 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86a3e0a-80ae-479e-b3c8-fcd9aa452640 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:13.329539 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86a3e0a-80ae-479e-b3c8-fcd9aa452640 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:13.330195 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86a3e0a-80ae-479e-b3c8-fcd9aa452640 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:13.342371 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0719f25-75ae-4796-a326-851048cf8ec2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:13.343831 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0719f25-75ae-4796-a326-851048cf8ec2 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:13.344123 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0719f25-75ae-4796-a326-851048cf8ec2 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:13.344417 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0719f25-75ae-4796-a326-851048cf8ec2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:13.348515 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-b86a3e0a-80ae-479e-b3c8-fcd9aa452640 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:13.349693 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-b86a3e0a-80ae-479e-b3c8-fcd9aa452640 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:13.420639 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0719f25-75ae-4796-a326-851048cf8ec2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:13.426639 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 337/661] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:20:13.426639 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86a3e0a-80ae-479e-b3c8-fcd9aa452640 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:13.426878 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0719f25-75ae-4796-a326-851048cf8ec2 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:13.518087 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:13.537630 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:13.542035 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86a3e0a-80ae-479e-b3c8-fcd9aa452640 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:13.547270 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 325/662] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:20:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 12 17:20:13.547270 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0719f25-75ae-4796-a326-851048cf8ec2 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:13.592378 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae714b9a-2fc4-4c1c-b568-b950aaa704bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:13.593216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae714b9a-2fc4-4c1c-b568-b950aaa704bf None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:13.593660 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae714b9a-2fc4-4c1c-b568-b950aaa704bf None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:13.594021 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae714b9a-2fc4-4c1c-b568-b950aaa704bf None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:13.594790 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae714b9a-2fc4-4c1c-b568-b950aaa704bf None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:13.595100 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae714b9a-2fc4-4c1c-b568-b950aaa704bf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:13.596464 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae714b9a-2fc4-4c1c-b568-b950aaa704bf None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:13.609907 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 338/663] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:13.611568 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae714b9a-2fc4-4c1c-b568-b950aaa704bf None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:13.658644 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0719f25-75ae-4796-a326-851048cf8ec2 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:13.694622 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b94eed-d361-4c96-a4cf-19c70934987b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:13.695293 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b94eed-d361-4c96-a4cf-19c70934987b None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:13.695586 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b94eed-d361-4c96-a4cf-19c70934987b None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:13.695868 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b94eed-d361-4c96-a4cf-19c70934987b None admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:13.696709 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b94eed-d361-4c96-a4cf-19c70934987b None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:13.697012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b94eed-d361-4c96-a4cf-19c70934987b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:13.698466 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b94eed-d361-4c96-a4cf-19c70934987b None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:13.716851 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 326/664] 10.222.0.16 () {56 vars in 1137 bytes} [Tue Jul 12 17:20:13 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:13.717678 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b94eed-d361-4c96-a4cf-19c70934987b None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:13.736725 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae714b9a-2fc4-4c1c-b568-b950aaa704bf None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:13.776759 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35b703c-6ce0-4cd2-8c9e-12e9c664302e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:13.777145 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:13.777583 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35b703c-6ce0-4cd2-8c9e-12e9c664302e None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:13.777844 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35b703c-6ce0-4cd2-8c9e-12e9c664302e None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:13.778123 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35b703c-6ce0-4cd2-8c9e-12e9c664302e None admin] PATH_INFO: `/v3/roles/service` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:13.781810 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:13.790198 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c35b703c-6ce0-4cd2-8c9e-12e9c664302e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:20:13.798530 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-c35b703c-6ce0-4cd2-8c9e-12e9c664302e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:20:13.798530 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:13.798530 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:13.798530 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:13.798530 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:13.798530 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:13.798530 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:13.798530 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:13.798530 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:13.798530 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:13.798530 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:13.798530 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:13.798530 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:20:13.798530 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:20:13.798530 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:20:13.798530 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:20:13.799673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:20:13.799673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:20:13.799673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:20:13.799673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:20:13.799673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:13.799673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:13.799673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:13.799673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:13.799673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:13.799673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:13.799673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:13.799673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:13.799673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:13.799673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:13.799673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:13.799673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:13.800718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:13.800718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:13.800718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:13.800718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:13.800718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:20:13.800718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:20:13.800718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:20:13.800718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:20:13.800718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:20:13.800718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:20:13.800718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:20:13.800718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:13.800718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 339/665] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Jul 12 17:20:13 2022] GET /identity/v3/roles/service => generated 85 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:13.801591 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35b703c-6ce0-4cd2-8c9e-12e9c664302e None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:13.829176 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b94eed-d361-4c96-a4cf-19c70934987b None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:13.866527 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e0d5ff-3f24-4393-8f85-3c6df067bb33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:13.867321 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e0d5ff-3f24-4393-8f85-3c6df067bb33 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:13.867614 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e0d5ff-3f24-4393-8f85-3c6df067bb33 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:13.867894 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e0d5ff-3f24-4393-8f85-3c6df067bb33 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:13.878217 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e0d5ff-3f24-4393-8f85-3c6df067bb33 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:13.879708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e0d5ff-3f24-4393-8f85-3c6df067bb33 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:13.897755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-66e0d5ff-3f24-4393-8f85-3c6df067bb33 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:13.897755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:13.897755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:13.897755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:13.897755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:13.897755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:13.897755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:13.897755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:13.897755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:13.897755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:13.897755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:13.897755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:13.897755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:13.897755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:13.897755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:13.898823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:13.898823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:13.898823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:13.898823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:13.898823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:13.898823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:13.898823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:13.898823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:13.898823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:13.898823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:13.898823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:13.898823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:13.898823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:13.898823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:13.898823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:13.898823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:13.898823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:13.900027 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:13.900027 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:13.900027 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:13.900027 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:13.903493 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 327/666] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:13.904369 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e0d5ff-3f24-4393-8f85-3c6df067bb33 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:13.910517 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35b703c-6ce0-4cd2-8c9e-12e9c664302e None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:13.946552 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df506c0-4f60-4866-ba06-be02cf6af55f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:13.947347 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df506c0-4f60-4866-ba06-be02cf6af55f None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:13.947635 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df506c0-4f60-4866-ba06-be02cf6af55f None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:13.947914 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df506c0-4f60-4866-ba06-be02cf6af55f None admin] PATH_INFO: `/v3/projects/service` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:13.956199 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df506c0-4f60-4866-ba06-be02cf6af55f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:13.957908 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df506c0-4f60-4866-ba06-be02cf6af55f None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:13.973536 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-7df506c0-4f60-4866-ba06-be02cf6af55f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:13.973536 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:13.973536 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:13.973536 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:13.973536 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:13.973536 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:13.973536 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:13.973536 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:13.973536 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:13.973536 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:13.973536 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:13.973536 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:13.973536 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:20:13.973536 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:20:13.973536 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:20:13.974771 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:20:13.974771 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:13.974771 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:13.974771 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:13.974771 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:13.974771 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:13.974771 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:13.974771 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:13.974771 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:13.974771 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:13.974771 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:13.974771 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:13.974771 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:13.974771 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:13.974771 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:13.974771 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:13.974771 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:13.975772 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:20:13.975772 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:20:13.975772 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:20:13.975772 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:20:13.975772 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:20:13.975772 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:20:13.975772 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:13.975772 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:13.979869 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 340/667] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:20:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:13.980818 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df506c0-4f60-4866-ba06-be02cf6af55f None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:14.016698 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e0d5ff-3f24-4393-8f85-3c6df067bb33 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:14.027249 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:14.041645 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:14.054388 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405652cb-d1bb-43ed-ac9d-1f4cd9ba0587 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:14.055015 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405652cb-d1bb-43ed-ac9d-1f4cd9ba0587 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:14.055306 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405652cb-d1bb-43ed-ac9d-1f4cd9ba0587 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:14.055594 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405652cb-d1bb-43ed-ac9d-1f4cd9ba0587 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:14.095178 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df506c0-4f60-4866-ba06-be02cf6af55f None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:14.131342 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405652cb-d1bb-43ed-ac9d-1f4cd9ba0587 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:14.132849 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405652cb-d1bb-43ed-ac9d-1f4cd9ba0587 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:14.133874 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbfaf22b-3ebc-4a3a-a70f-463032f9bb6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:14.134670 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfaf22b-3ebc-4a3a-a70f-463032f9bb6a None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:14.135123 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfaf22b-3ebc-4a3a-a70f-463032f9bb6a None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:14.135467 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfaf22b-3ebc-4a3a-a70f-463032f9bb6a None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:14.136426 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbfaf22b-3ebc-4a3a-a70f-463032f9bb6a None admin] RBAC: Adding query filter params (name=service) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:14.136815 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbfaf22b-3ebc-4a3a-a70f-463032f9bb6a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:14.138273 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbfaf22b-3ebc-4a3a-a70f-463032f9bb6a None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:14.152968 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 341/668] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Jul 12 17:20:13 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:14.152968 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbfaf22b-3ebc-4a3a-a70f-463032f9bb6a None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:14.248701 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 328/669] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:20:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 12 17:20:14.248701 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405652cb-d1bb-43ed-ac9d-1f4cd9ba0587 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:14.257818 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:14.267002 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbfaf22b-3ebc-4a3a-a70f-463032f9bb6a None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:14.286059 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:14.303622 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d943b734-e6ab-4bda-8a94-c23fbf24e4ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:14.304490 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d943b734-e6ab-4bda-8a94-c23fbf24e4ca None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:14.304828 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d943b734-e6ab-4bda-8a94-c23fbf24e4ca None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:14.305192 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d943b734-e6ab-4bda-8a94-c23fbf24e4ca None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:14.306009 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d943b734-e6ab-4bda-8a94-c23fbf24e4ca None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:14.306423 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d943b734-e6ab-4bda-8a94-c23fbf24e4ca None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:14.307824 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d943b734-e6ab-4bda-8a94-c23fbf24e4ca None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:14.322596 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 342/670] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:14.323502 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d943b734-e6ab-4bda-8a94-c23fbf24e4ca None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:14.368546 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405652cb-d1bb-43ed-ac9d-1f4cd9ba0587 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:14.408918 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e59d5e0-e850-4c11-8fb4-67f2b69e9f51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:14.409763 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e59d5e0-e850-4c11-8fb4-67f2b69e9f51 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:14.410158 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e59d5e0-e850-4c11-8fb4-67f2b69e9f51 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:14.410579 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e59d5e0-e850-4c11-8fb4-67f2b69e9f51 None admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:14.411494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e59d5e0-e850-4c11-8fb4-67f2b69e9f51 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:14.411964 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e59d5e0-e850-4c11-8fb4-67f2b69e9f51 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:14.413564 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e59d5e0-e850-4c11-8fb4-67f2b69e9f51 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:14.427404 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 329/671] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Jul 12 17:20:14 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:14.428673 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e59d5e0-e850-4c11-8fb4-67f2b69e9f51 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:14.433012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d943b734-e6ab-4bda-8a94-c23fbf24e4ca None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:14.473395 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ee7fef-2a4d-4012-8334-eee873dba8ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:14.474374 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ee7fef-2a4d-4012-8334-eee873dba8ce None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:14.474764 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ee7fef-2a4d-4012-8334-eee873dba8ce None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:14.475059 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ee7fef-2a4d-4012-8334-eee873dba8ce None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:14.487212 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64ee7fef-2a4d-4012-8334-eee873dba8ce None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:14.488667 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64ee7fef-2a4d-4012-8334-eee873dba8ce None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:14.504174 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:14.507477 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-64ee7fef-2a4d-4012-8334-eee873dba8ce None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:14.507477 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:14.507477 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:14.507477 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:14.507477 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:14.507477 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:14.507477 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:14.507477 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:14.507477 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:14.507477 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:14.507477 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:14.507477 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:14.507477 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:14.507477 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:14.507477 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:14.508379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:14.508379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:14.508379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:14.508379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:14.508379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:14.508379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:14.508379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:14.508379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:14.508379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:14.508379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:14.508379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:14.508379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:14.508379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:14.508379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:14.508379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:14.508379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:14.508379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:14.509608 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:14.509608 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:14.509608 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:14.509608 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:14.509608 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 343/672] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:14.510281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ee7fef-2a4d-4012-8334-eee873dba8ce None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:14.542272 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:14.545964 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e59d5e0-e850-4c11-8fb4-67f2b69e9f51 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:14.587122 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09b9b82-9b28-4e01-969c-a7e72d6e25c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:14.587964 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09b9b82-9b28-4e01-969c-a7e72d6e25c3 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:14.588270 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09b9b82-9b28-4e01-969c-a7e72d6e25c3 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:14.588552 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09b9b82-9b28-4e01-969c-a7e72d6e25c3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:14.600634 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09b9b82-9b28-4e01-969c-a7e72d6e25c3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:14.602095 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09b9b82-9b28-4e01-969c-a7e72d6e25c3 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:14.621926 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-e09b9b82-9b28-4e01-969c-a7e72d6e25c3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:14.621926 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:14.621926 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:14.621926 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:14.621926 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:14.621926 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:14.621926 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:14.621926 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:14.621926 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:14.621926 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:14.621926 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:14.621926 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:14.621926 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:14.621926 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:14.621926 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:14.623125 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:14.623125 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:14.623125 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:14.623125 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:14.623125 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:14.623125 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:14.623125 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:14.623125 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:14.623125 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:14.623125 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:14.623125 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:14.623125 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:14.623125 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:14.623125 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:14.623125 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:14.623125 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:14.623125 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:14.624889 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:14.624889 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:14.624889 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:14.624889 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:14.624889 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 330/673] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:14.625186 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09b9b82-9b28-4e01-969c-a7e72d6e25c3 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:14.633604 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ee7fef-2a4d-4012-8334-eee873dba8ce None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:14.672500 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-958de097-1136-4d81-890d-c3f8831d009a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:14.673433 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-958de097-1136-4d81-890d-c3f8831d009a None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:14.673828 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-958de097-1136-4d81-890d-c3f8831d009a None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:14.674178 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-958de097-1136-4d81-890d-c3f8831d009a None admin] PATH_INFO: `/v3/projects/service` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:14.682351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-958de097-1136-4d81-890d-c3f8831d009a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:14.683731 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-958de097-1136-4d81-890d-c3f8831d009a None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:14.700008 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-958de097-1136-4d81-890d-c3f8831d009a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:14.700008 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:14.700008 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:14.700008 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:14.700008 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:14.700008 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:14.700008 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:14.700008 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:14.700008 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:14.700008 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:14.700008 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:14.700008 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:14.700008 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:20:14.700008 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:20:14.700008 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:20:14.701164 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:20:14.701164 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:14.701164 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:14.701164 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:14.701164 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:14.701164 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:14.701164 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:14.701164 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:14.701164 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:14.701164 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:14.701164 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:14.701164 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:14.701164 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:14.701164 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:14.701164 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:14.701164 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:14.701164 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:14.702127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:20:14.702127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:20:14.702127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:20:14.702127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:20:14.702127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:20:14.702127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:20:14.702127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:14.702127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:14.702127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 344/674] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:20:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:14.702803 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-958de097-1136-4d81-890d-c3f8831d009a None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:14.736397 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09b9b82-9b28-4e01-969c-a7e72d6e25c3 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:14.757715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:14.773311 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e9e316-8920-4a5b-8ead-595344909e48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:14.774064 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e9e316-8920-4a5b-8ead-595344909e48 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:14.774515 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e9e316-8920-4a5b-8ead-595344909e48 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:14.774871 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e9e316-8920-4a5b-8ead-595344909e48 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:14.775905 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e9e316-8920-4a5b-8ead-595344909e48 None admin] RBAC: Adding query filter params (role.id=bef74d4975164f75bcec96ac47afea73, scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2, user.id=4018d94a1feb4d6dade88ee9663f9729) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:14.776341 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e9e316-8920-4a5b-8ead-595344909e48 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:14.777866 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e9e316-8920-4a5b-8ead-595344909e48 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:14.794455 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 331/675] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:20:14 2022] GET /identity/v3/role_assignments?user.id=4018d94a1feb4d6dade88ee9663f9729&scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2&role.id=bef74d4975164f75bcec96ac47afea73 => generated 608 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:14.794455 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e9e316-8920-4a5b-8ead-595344909e48 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:14.815741 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:14.827189 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-958de097-1136-4d81-890d-c3f8831d009a None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:14.881961 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34afe0fe-86a1-4864-9b38-f9f59b29a7ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:14.882920 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34afe0fe-86a1-4864-9b38-f9f59b29a7ce None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:14.883212 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34afe0fe-86a1-4864-9b38-f9f59b29a7ce None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:14.883498 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34afe0fe-86a1-4864-9b38-f9f59b29a7ce None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:14.884803 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34afe0fe-86a1-4864-9b38-f9f59b29a7ce None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:14.885421 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34afe0fe-86a1-4864-9b38-f9f59b29a7ce None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:14.887173 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34afe0fe-86a1-4864-9b38-f9f59b29a7ce None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:14.904807 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 345/676] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:14.906411 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34afe0fe-86a1-4864-9b38-f9f59b29a7ce None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:14.934160 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e9e316-8920-4a5b-8ead-595344909e48 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:14.978274 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a2b0e9-873f-4699-abb4-72632c1c43e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:14.979471 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a2b0e9-873f-4699-abb4-72632c1c43e3 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:14.979471 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a2b0e9-873f-4699-abb4-72632c1c43e3 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:14.982840 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a2b0e9-873f-4699-abb4-72632c1c43e3 None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:14.982840 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64a2b0e9-873f-4699-abb4-72632c1c43e3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:14.982840 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64a2b0e9-873f-4699-abb4-72632c1c43e3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:14.982840 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64a2b0e9-873f-4699-abb4-72632c1c43e3 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:14.998059 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 332/677] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:14.999695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a2b0e9-873f-4699-abb4-72632c1c43e3 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:15.046570 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34afe0fe-86a1-4864-9b38-f9f59b29a7ce None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:15.057880 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:15.088962 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c99af4-cbc4-4623-ab59-4507e5b172b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:15.089637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c99af4-cbc4-4623-ab59-4507e5b172b5 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:15.089928 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c99af4-cbc4-4623-ab59-4507e5b172b5 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:15.090211 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c99af4-cbc4-4623-ab59-4507e5b172b5 None admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:15.091034 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c99af4-cbc4-4623-ab59-4507e5b172b5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:15.091350 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c99af4-cbc4-4623-ab59-4507e5b172b5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:15.092850 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c99af4-cbc4-4623-ab59-4507e5b172b5 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:15.114455 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 346/678] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Jul 12 17:20:14 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:15.114455 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c99af4-cbc4-4623-ab59-4507e5b172b5 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:15.119254 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a2b0e9-873f-4699-abb4-72632c1c43e3 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:15.125173 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:15.160846 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddaea8c1-0221-416b-b67d-e31a25328683 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:15.161566 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddaea8c1-0221-416b-b67d-e31a25328683 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:15.161854 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddaea8c1-0221-416b-b67d-e31a25328683 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:15.162349 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddaea8c1-0221-416b-b67d-e31a25328683 None admin] PATH_INFO: `/v3/users/swift` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:15.169823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddaea8c1-0221-416b-b67d-e31a25328683 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:15.171436 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddaea8c1-0221-416b-b67d-e31a25328683 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:15.186106 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-ddaea8c1-0221-416b-b67d-e31a25328683 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 12 17:20:15.186106 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:15.186106 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:15.186106 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:15.186106 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:15.186106 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:15.186106 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:15.186106 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:15.186106 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:15.186106 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:15.186106 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:15.186106 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:15.186106 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:20:15.186106 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:20:15.186106 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:20:15.186106 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:20:15.187737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:15.187737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:15.187737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:20:15.187737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:15.187737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:20:15.187737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:15.187737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:15.187737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:15.187737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:15.187737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:15.187737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:15.187737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:15.187737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:15.187737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:15.187737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:15.187737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:15.187737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:15.187737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:15.188712 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:15.188712 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:15.188712 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:20:15.188712 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:20:15.188712 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:20:15.188712 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:20:15.188712 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:20:15.188712 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:20:15.188712 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 12 17:20:15.188712 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:15.188712 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 333/679] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Jul 12 17:20:14 2022] GET /identity/v3/users/swift => generated 83 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:15.190700 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb389c7-4992-4de2-bcb8-bbafc1a009b5 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:15.190700 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb389c7-4992-4de2-bcb8-bbafc1a009b5 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:15.190700 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb389c7-4992-4de2-bcb8-bbafc1a009b5 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:15.195481 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 334/680] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:15.195481 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb389c7-4992-4de2-bcb8-bbafc1a009b5 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:15.226654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c99af4-cbc4-4623-ab59-4507e5b172b5 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:15.265428 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a50a03-e930-4bcb-a29a-77be29cc3bf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:15.266231 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a50a03-e930-4bcb-a29a-77be29cc3bf1 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:15.266555 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a50a03-e930-4bcb-a29a-77be29cc3bf1 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:15.266841 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a50a03-e930-4bcb-a29a-77be29cc3bf1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:15.273806 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a50a03-e930-4bcb-a29a-77be29cc3bf1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:15.275271 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a50a03-e930-4bcb-a29a-77be29cc3bf1 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:15.292253 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-26a50a03-e930-4bcb-a29a-77be29cc3bf1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:15.292253 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:15.292253 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:15.292253 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:15.292253 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:15.292253 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:15.292253 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:15.292253 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:15.292253 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:15.292253 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:15.292253 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:15.292253 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:15.292253 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:20:15.292253 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:20:15.292253 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:20:15.293251 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:20:15.293251 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:15.293251 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:15.293251 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:15.293251 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:15.293251 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:15.293251 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:15.293251 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:15.293251 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:15.293251 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:15.293251 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:15.293251 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:15.293251 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:15.293251 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:15.293251 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:15.293251 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:15.293251 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:15.294135 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:20:15.294135 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:20:15.294135 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:20:15.294135 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:20:15.294135 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:20:15.294135 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:20:15.294135 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:15.294135 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:15.298945 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 347/681] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:20:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:15.299660 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a50a03-e930-4bcb-a29a-77be29cc3bf1 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:15.308441 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb389c7-4992-4de2-bcb8-bbafc1a009b5 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:15.312497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:15.347117 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81266d2b-53f1-40b1-afc5-a5b6db707c6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:15.347783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81266d2b-53f1-40b1-afc5-a5b6db707c6e None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:15.348077 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81266d2b-53f1-40b1-afc5-a5b6db707c6e None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:15.348361 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81266d2b-53f1-40b1-afc5-a5b6db707c6e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:15.349254 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81266d2b-53f1-40b1-afc5-a5b6db707c6e None admin] RBAC: Adding query filter params (role.id=bef74d4975164f75bcec96ac47afea73, scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2, user.id=bf5bf02ae13d4d86a1b157369ef07a00) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:15.349576 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81266d2b-53f1-40b1-afc5-a5b6db707c6e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:15.351057 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81266d2b-53f1-40b1-afc5-a5b6db707c6e None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:15.362854 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 335/682] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:20:15 2022] GET /identity/v3/role_assignments?user.id=bf5bf02ae13d4d86a1b157369ef07a00&scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2&role.id=bef74d4975164f75bcec96ac47afea73 => generated 608 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:15.365698 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9a371e-2dca-4b9f-86e2-063f049370b8 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:15.366342 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9a371e-2dca-4b9f-86e2-063f049370b8 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:15.367433 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9a371e-2dca-4b9f-86e2-063f049370b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:15.382149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:15.397513 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-ba9a371e-2dca-4b9f-86e2-063f049370b8 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:15.399167 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-ba9a371e-2dca-4b9f-86e2-063f049370b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:15.429127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a50a03-e930-4bcb-a29a-77be29cc3bf1 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:15.477444 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400803b0-7fa4-410c-922f-a98da5b02f9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:15.478397 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400803b0-7fa4-410c-922f-a98da5b02f9f None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:15.478716 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400803b0-7fa4-410c-922f-a98da5b02f9f None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:15.479002 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400803b0-7fa4-410c-922f-a98da5b02f9f None admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:15.480279 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400803b0-7fa4-410c-922f-a98da5b02f9f None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:15.480961 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400803b0-7fa4-410c-922f-a98da5b02f9f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:15.482698 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400803b0-7fa4-410c-922f-a98da5b02f9f None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:15.493490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 336/683] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:20:15.495038 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9a371e-2dca-4b9f-86e2-063f049370b8 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:15.507288 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 348/684] 10.222.0.16 () {56 vars in 1135 bytes} [Tue Jul 12 17:20:15 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:15.515569 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400803b0-7fa4-410c-922f-a98da5b02f9f None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:15.638588 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:15.642422 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9a371e-2dca-4b9f-86e2-063f049370b8 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:15.648126 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400803b0-7fa4-410c-922f-a98da5b02f9f None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:15.679732 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0c87a1-d08e-4754-904b-31126c713cea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:15.680432 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0c87a1-d08e-4754-904b-31126c713cea None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:15.680778 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0c87a1-d08e-4754-904b-31126c713cea None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:15.681123 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0c87a1-d08e-4754-904b-31126c713cea None admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:15.681285 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:15.682049 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c0c87a1-d08e-4754-904b-31126c713cea None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:15.682446 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c0c87a1-d08e-4754-904b-31126c713cea None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:15.683980 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c0c87a1-d08e-4754-904b-31126c713cea None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:15.690360 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78882b5-531e-45b2-89b5-c1b312f69373 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:15.691070 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78882b5-531e-45b2-89b5-c1b312f69373 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:15.691354 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78882b5-531e-45b2-89b5-c1b312f69373 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:15.691632 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f78882b5-531e-45b2-89b5-c1b312f69373 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:15.698310 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 337/685] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Jul 12 17:20:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:15.703975 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0c87a1-d08e-4754-904b-31126c713cea None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:15.763649 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f78882b5-531e-45b2-89b5-c1b312f69373 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:15.765868 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f78882b5-531e-45b2-89b5-c1b312f69373 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:15.811055 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0c87a1-d08e-4754-904b-31126c713cea None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:15.848340 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705ed4d6-79a7-49d0-99e7-80030439dee1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:15.849395 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705ed4d6-79a7-49d0-99e7-80030439dee1 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:15.849759 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705ed4d6-79a7-49d0-99e7-80030439dee1 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:15.850469 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705ed4d6-79a7-49d0-99e7-80030439dee1 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:15.864235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705ed4d6-79a7-49d0-99e7-80030439dee1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:20:15.869957 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 349/686] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:20:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 355 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 12 17:20:15.870995 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78882b5-531e-45b2-89b5-c1b312f69373 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:15.877411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-705ed4d6-79a7-49d0-99e7-80030439dee1 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:20:15.877411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:15.877411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:15.877411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:15.877411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:15.877411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:15.877411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:15.877411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:15.877411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:15.877411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:15.877411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:15.877411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:15.877411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:20:15.877411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:20:15.877411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:20:15.877411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:20:15.878387 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:20:15.878387 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:20:15.878387 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:20:15.878387 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:20:15.878387 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:15.878387 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:15.878387 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:15.878387 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:15.878387 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:15.878387 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:15.878387 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:15.878387 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:15.878387 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:15.878387 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:15.878387 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:15.878387 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:15.879282 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:15.879282 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:15.879282 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:15.879282 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:15.879282 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:20:15.879282 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:20:15.879282 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:20:15.879282 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:20:15.879282 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:20:15.879282 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:20:15.879282 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:20:15.879282 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:15.879923 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 338/687] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Jul 12 17:20:15 2022] GET /identity/v3/roles/admin => generated 83 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:15.880874 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705ed4d6-79a7-49d0-99e7-80030439dee1 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:15.907553 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:15.908968 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:15.990958 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f78882b5-531e-45b2-89b5-c1b312f69373 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:16.013104 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705ed4d6-79a7-49d0-99e7-80030439dee1 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:16.031345 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb698bf-b282-4676-a66f-806d28dd05cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:16.031820 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb698bf-b282-4676-a66f-806d28dd05cd None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:16.032110 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb698bf-b282-4676-a66f-806d28dd05cd None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:16.032392 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb698bf-b282-4676-a66f-806d28dd05cd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:16.033245 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb698bf-b282-4676-a66f-806d28dd05cd None admin] RBAC: Adding query filter params (role.id=bef74d4975164f75bcec96ac47afea73, scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2, user.id=f6a82b49464846d1a207189f17fd4cf7) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:16.033632 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb698bf-b282-4676-a66f-806d28dd05cd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:16.035121 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb698bf-b282-4676-a66f-806d28dd05cd None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:16.054501 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 350/688] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:20:15 2022] GET /identity/v3/role_assignments?user.id=f6a82b49464846d1a207189f17fd4cf7&scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2&role.id=bef74d4975164f75bcec96ac47afea73 => generated 608 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:16.054501 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb698bf-b282-4676-a66f-806d28dd05cd None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:16.094105 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2265cb89-631e-465f-97df-d933a19c2f9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:16.094954 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2265cb89-631e-465f-97df-d933a19c2f9b None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:16.095249 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2265cb89-631e-465f-97df-d933a19c2f9b None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:16.095533 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2265cb89-631e-465f-97df-d933a19c2f9b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:16.125389 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2265cb89-631e-465f-97df-d933a19c2f9b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:16.127041 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2265cb89-631e-465f-97df-d933a19c2f9b None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:16.170943 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:16.172713 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-2265cb89-631e-465f-97df-d933a19c2f9b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:16.172713 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:16.172713 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:16.172713 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:16.172713 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:16.172713 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:16.172713 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:16.172713 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:16.172713 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:16.172713 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:16.172713 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:16.172713 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:16.172713 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:16.172713 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:16.172713 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:16.173798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:16.173798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:16.173798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:16.173798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:16.173798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:16.173798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:16.173798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:16.173798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:16.173798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:16.173798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:16.173798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:16.173798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:16.173798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:16.173798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:16.173798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:16.173798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:16.173798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:16.175023 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:16.175023 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:16.175023 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:16.175023 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:16.175023 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 339/689] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 294 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:16.180000 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2265cb89-631e-465f-97df-d933a19c2f9b None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:16.181378 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb698bf-b282-4676-a66f-806d28dd05cd None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:16.265643 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4af551-8001-40a1-bd4b-329dc166da8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:16.266267 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4af551-8001-40a1-bd4b-329dc166da8a None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:16.266594 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4af551-8001-40a1-bd4b-329dc166da8a None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:16.266879 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4af551-8001-40a1-bd4b-329dc166da8a None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:16.267651 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a4af551-8001-40a1-bd4b-329dc166da8a None admin] RBAC: Adding query filter params (name=admin) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:16.267969 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a4af551-8001-40a1-bd4b-329dc166da8a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:16.269509 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a4af551-8001-40a1-bd4b-329dc166da8a None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:16.279691 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 351/690] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:20:16 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:16.288362 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4af551-8001-40a1-bd4b-329dc166da8a None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:16.308923 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:16.341371 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2265cb89-631e-465f-97df-d933a19c2f9b None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:16.392728 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70bbad80-0228-4b2a-bd1e-7bdd5f3788dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:16.393425 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bbad80-0228-4b2a-bd1e-7bdd5f3788dc None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:16.393742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bbad80-0228-4b2a-bd1e-7bdd5f3788dc None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:16.394030 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bbad80-0228-4b2a-bd1e-7bdd5f3788dc None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:16.394979 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70bbad80-0228-4b2a-bd1e-7bdd5f3788dc None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:16.395302 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70bbad80-0228-4b2a-bd1e-7bdd5f3788dc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:16.396693 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70bbad80-0228-4b2a-bd1e-7bdd5f3788dc None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:16.417576 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 340/691] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:16.423737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70bbad80-0228-4b2a-bd1e-7bdd5f3788dc None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:16.447656 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4af551-8001-40a1-bd4b-329dc166da8a None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:16.484327 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1fd00a6-2c70-4a3e-8009-9f9b5fb7fd75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:16.485135 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fd00a6-2c70-4a3e-8009-9f9b5fb7fd75 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:16.485427 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fd00a6-2c70-4a3e-8009-9f9b5fb7fd75 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:16.485708 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1fd00a6-2c70-4a3e-8009-9f9b5fb7fd75 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:16.497106 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1fd00a6-2c70-4a3e-8009-9f9b5fb7fd75 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:16.498690 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1fd00a6-2c70-4a3e-8009-9f9b5fb7fd75 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:16.516818 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-c1fd00a6-2c70-4a3e-8009-9f9b5fb7fd75 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:16.516818 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:16.516818 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:16.516818 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:16.516818 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:16.516818 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:16.516818 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:16.516818 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:16.516818 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:16.516818 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:16.516818 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:16.516818 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:16.516818 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:16.516818 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:16.516818 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:16.517988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:16.517988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:16.517988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:16.517988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:16.517988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:16.517988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:16.517988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:16.517988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:16.517988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:16.517988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:16.517988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:16.517988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:16.517988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:16.517988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:16.517988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:16.517988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:16.517988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:16.519205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:16.519205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:16.519205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:16.519205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:16.519205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 352/692] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:20:16.527652 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1fd00a6-2c70-4a3e-8009-9f9b5fb7fd75 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:16.550156 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:16.586079 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70bbad80-0228-4b2a-bd1e-7bdd5f3788dc None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:16.653888 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:16.662790 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e4f063-7072-456b-a9d2-9ec9a199de9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:16.663597 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e4f063-7072-456b-a9d2-9ec9a199de9f None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:16.663885 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e4f063-7072-456b-a9d2-9ec9a199de9f None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:16.664164 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e4f063-7072-456b-a9d2-9ec9a199de9f None admin] PATH_INFO: `/v3/projects/service` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:16.677858 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e4f063-7072-456b-a9d2-9ec9a199de9f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:16.679341 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e4f063-7072-456b-a9d2-9ec9a199de9f None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:16.713021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-11e4f063-7072-456b-a9d2-9ec9a199de9f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:16.713021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:16.713021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:16.713021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:16.713021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:16.713021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:16.713021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:16.713021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:16.713021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:16.713021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:16.713021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:16.713021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:16.713021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:20:16.713021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:20:16.713021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:20:16.713964 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:20:16.713964 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:16.713964 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:16.713964 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:16.713964 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:16.713964 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:16.713964 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:16.713964 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:16.713964 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:16.713964 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:16.713964 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:16.713964 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:16.713964 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:16.713964 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:16.713964 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:16.713964 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:16.713964 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:16.714899 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:20:16.714899 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:20:16.714899 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:20:16.714899 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:20:16.714899 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:20:16.714899 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:20:16.714899 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:16.714899 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:16.715330 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1fd00a6-2c70-4a3e-8009-9f9b5fb7fd75 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:16.718082 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 341/693] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:20:16 2022] GET /identity/v3/projects/service => generated 88 bytes in 294 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:16.723451 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e4f063-7072-456b-a9d2-9ec9a199de9f None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:16.758571 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e2be06-d58d-428a-a2af-0490bd04266a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:16.759612 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e2be06-d58d-428a-a2af-0490bd04266a None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:16.761292 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e2be06-d58d-428a-a2af-0490bd04266a None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:16.761801 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e2be06-d58d-428a-a2af-0490bd04266a None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:16.763189 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e2be06-d58d-428a-a2af-0490bd04266a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:16.764995 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e2be06-d58d-428a-a2af-0490bd04266a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:16.769288 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61e2be06-d58d-428a-a2af-0490bd04266a None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:16.810015 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 353/694] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:20:16.815455 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e2be06-d58d-428a-a2af-0490bd04266a None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:16.884927 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:16.891996 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e4f063-7072-456b-a9d2-9ec9a199de9f None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:16.923271 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e2be06-d58d-428a-a2af-0490bd04266a None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:16.931257 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54df9af7-857f-41f9-87bf-be39cebc2da0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:16.932271 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54df9af7-857f-41f9-87bf-be39cebc2da0 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:16.937079 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54df9af7-857f-41f9-87bf-be39cebc2da0 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:16.937079 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54df9af7-857f-41f9-87bf-be39cebc2da0 None admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:16.937079 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54df9af7-857f-41f9-87bf-be39cebc2da0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:16.937079 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54df9af7-857f-41f9-87bf-be39cebc2da0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:16.937079 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54df9af7-857f-41f9-87bf-be39cebc2da0 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:16.957743 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 342/695] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Jul 12 17:20:16 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:20:16.963340 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54df9af7-857f-41f9-87bf-be39cebc2da0 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:16.965126 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ade6ee-5647-49b3-a214-33266fedf97d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:16.965954 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ade6ee-5647-49b3-a214-33266fedf97d None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:16.966243 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ade6ee-5647-49b3-a214-33266fedf97d None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:16.966555 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ade6ee-5647-49b3-a214-33266fedf97d None admin] PATH_INFO: `/v3/users/nova` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:16.987340 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58ade6ee-5647-49b3-a214-33266fedf97d None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:16.988936 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58ade6ee-5647-49b3-a214-33266fedf97d None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:17.018351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-58ade6ee-5647-49b3-a214-33266fedf97d None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 12 17:20:17.018351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:17.018351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:17.018351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:17.018351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:17.018351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:17.018351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:17.018351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:17.018351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:17.018351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:17.018351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:17.018351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:17.018351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:20:17.018351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:20:17.018351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:20:17.018351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:20:17.019457 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:17.019457 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:17.019457 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:20:17.019457 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:17.019457 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:20:17.019457 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:17.019457 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:17.019457 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:17.019457 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:17.019457 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:17.019457 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:17.019457 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:17.019457 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:17.019457 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:17.019457 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:17.019457 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:17.019457 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:17.019457 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:17.021140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:17.021140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:17.021140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:20:17.021140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:20:17.021140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:20:17.021140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:20:17.021140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:20:17.021140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:20:17.021140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 12 17:20:17.021140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:17.021140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 354/696] 10.222.0.16 () {56 vars in 1093 bytes} [Tue Jul 12 17:20:16 2022] GET /identity/v3/users/nova => generated 82 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:17.025365 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ade6ee-5647-49b3-a214-33266fedf97d None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:17.080811 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:17.189811 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54df9af7-857f-41f9-87bf-be39cebc2da0 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:17.228196 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ade6ee-5647-49b3-a214-33266fedf97d None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:17.237876 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546276cb-3c49-4579-ba4c-19d86a95e877 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:17.239423 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546276cb-3c49-4579-ba4c-19d86a95e877 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:17.239721 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546276cb-3c49-4579-ba4c-19d86a95e877 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:17.240003 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546276cb-3c49-4579-ba4c-19d86a95e877 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:17.241331 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-546276cb-3c49-4579-ba4c-19d86a95e877 None admin] RBAC: Adding query filter params (role.id=bef74d4975164f75bcec96ac47afea73, scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2, user.id=d7906d1a19c642bd93507a27bf3e3dae) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:17.241660 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-546276cb-3c49-4579-ba4c-19d86a95e877 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:17.243740 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-546276cb-3c49-4579-ba4c-19d86a95e877 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:17.251734 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 343/697] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:20:16 2022] GET /identity/v3/role_assignments?user.id=d7906d1a19c642bd93507a27bf3e3dae&scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2&role.id=bef74d4975164f75bcec96ac47afea73 => generated 608 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:20:17.253619 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c0cc7a-8a9f-4447-8305-f83f616504fe None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:17.253916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c0cc7a-8a9f-4447-8305-f83f616504fe None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:17.254392 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c0cc7a-8a9f-4447-8305-f83f616504fe None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:17.256031 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 344/698] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:17.262472 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c6119c-b095-4586-a618-c030ce2f48c2 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:17.262771 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c6119c-b095-4586-a618-c030ce2f48c2 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:17.263037 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c6119c-b095-4586-a618-c030ce2f48c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:17.282387 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:17.301924 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-b4c6119c-b095-4586-a618-c030ce2f48c2 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:17.302266 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2018fae-4fbb-4010-9445-794fbedc687f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:17.302969 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2018fae-4fbb-4010-9445-794fbedc687f None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:17.303264 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2018fae-4fbb-4010-9445-794fbedc687f None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:17.303432 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-b4c6119c-b095-4586-a618-c030ce2f48c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:17.303588 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2018fae-4fbb-4010-9445-794fbedc687f None admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:17.304403 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2018fae-4fbb-4010-9445-794fbedc687f None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:17.304726 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2018fae-4fbb-4010-9445-794fbedc687f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:17.307814 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2018fae-4fbb-4010-9445-794fbedc687f None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:17.353093 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 355/699] 10.222.0.16 () {56 vars in 1133 bytes} [Tue Jul 12 17:20:17 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 328 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:17.359798 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2018fae-4fbb-4010-9445-794fbedc687f None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:17.451122 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 345/700] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:20:17.488731 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c6119c-b095-4586-a618-c030ce2f48c2 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:17.546919 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2018fae-4fbb-4010-9445-794fbedc687f None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:17.560602 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:17.588363 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8fd2424-c3c6-45e5-ae0d-72c9f5bbb52f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:17.589024 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fd2424-c3c6-45e5-ae0d-72c9f5bbb52f None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:17.589319 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fd2424-c3c6-45e5-ae0d-72c9f5bbb52f None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:17.589603 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8fd2424-c3c6-45e5-ae0d-72c9f5bbb52f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:17.650785 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c6119c-b095-4586-a618-c030ce2f48c2 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:17.668779 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:17.691740 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea6c406-522f-4a96-a708-e0d63c410717 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:17.692542 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea6c406-522f-4a96-a708-e0d63c410717 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:17.692829 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea6c406-522f-4a96-a708-e0d63c410717 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:17.692993 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8fd2424-c3c6-45e5-ae0d-72c9f5bbb52f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:17.693286 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea6c406-522f-4a96-a708-e0d63c410717 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:17.697353 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8fd2424-c3c6-45e5-ae0d-72c9f5bbb52f None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:17.707791 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea6c406-522f-4a96-a708-e0d63c410717 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:17.709018 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea6c406-522f-4a96-a708-e0d63c410717 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:17.730258 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-cea6c406-522f-4a96-a708-e0d63c410717 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:17.730258 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:17.730258 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:17.730258 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:17.730258 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:17.730258 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:17.730258 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:17.730258 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:17.730258 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:17.730258 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:17.730258 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:17.730258 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:17.730258 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:17.730258 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:17.730258 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:17.731279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:17.731279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:17.731279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:17.731279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:17.731279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:17.731279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:17.731279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:17.731279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:17.731279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:17.731279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:17.731279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:17.731279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:17.731279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:17.731279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:17.731279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:17.731279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:17.731279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:17.732162 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:17.732162 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:17.732162 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:17.732162 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:17.732162 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 346/701] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:20:17.740581 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea6c406-522f-4a96-a708-e0d63c410717 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:17.854393 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 356/702] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:20:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 495 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 12 17:20:17.855718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488d9c4d-7ff8-4d14-a5e6-65cb4f203baa None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:17.856015 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488d9c4d-7ff8-4d14-a5e6-65cb4f203baa None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:17.856305 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488d9c4d-7ff8-4d14-a5e6-65cb4f203baa None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:17.857904 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 357/703] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:17.863361 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488d9c4d-7ff8-4d14-a5e6-65cb4f203baa None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:17.891772 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:17.903229 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea6c406-522f-4a96-a708-e0d63c410717 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:17.944526 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3dc71f-d473-456a-baff-abb47cde7f41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:17.945119 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3dc71f-d473-456a-baff-abb47cde7f41 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:17.945456 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3dc71f-d473-456a-baff-abb47cde7f41 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:17.945820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3dc71f-d473-456a-baff-abb47cde7f41 None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:17.946808 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed3dc71f-d473-456a-baff-abb47cde7f41 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:17.947119 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed3dc71f-d473-456a-baff-abb47cde7f41 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:17.948622 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed3dc71f-d473-456a-baff-abb47cde7f41 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:17.962821 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 347/704] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:17.964217 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1799ad95-7bf5-406a-b96b-e927db2aa6f7 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:17.964487 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1799ad95-7bf5-406a-b96b-e927db2aa6f7 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:17.964726 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1799ad95-7bf5-406a-b96b-e927db2aa6f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:17.988858 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-1799ad95-7bf5-406a-b96b-e927db2aa6f7 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:17.990358 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-1799ad95-7bf5-406a-b96b-e927db2aa6f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:18.016325 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:18.035119 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488d9c4d-7ff8-4d14-a5e6-65cb4f203baa None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:18.072768 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a5cd51-df47-4af5-81f5-2607436ff757 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:18.073565 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a5cd51-df47-4af5-81f5-2607436ff757 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:18.073856 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a5cd51-df47-4af5-81f5-2607436ff757 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:18.074113 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a5cd51-df47-4af5-81f5-2607436ff757 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:18.085188 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 348/705] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:20:18.086394 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1799ad95-7bf5-406a-b96b-e927db2aa6f7 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:18.091373 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55a5cd51-df47-4af5-81f5-2607436ff757 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:18.093516 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55a5cd51-df47-4af5-81f5-2607436ff757 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:18.124811 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-55a5cd51-df47-4af5-81f5-2607436ff757 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:18.124811 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:18.124811 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:18.124811 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:18.124811 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:18.124811 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:18.124811 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:18.124811 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:18.124811 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:18.124811 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:18.124811 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:18.124811 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:18.124811 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:18.124811 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:18.124811 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:18.125740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:18.125740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:18.125740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:18.125740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:18.125740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:18.125740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:18.125740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:18.125740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:18.125740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:18.125740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:18.125740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:18.125740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:18.125740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:18.125740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:18.125740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:18.125740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:18.125740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:18.127091 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:18.127091 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:18.127091 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:18.127091 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:18.130828 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 358/706] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:18.131669 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a5cd51-df47-4af5-81f5-2607436ff757 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:18.199948 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:18.215397 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1799ad95-7bf5-406a-b96b-e927db2aa6f7 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:18.259512 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a5cd51-df47-4af5-81f5-2607436ff757 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:18.264120 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223c704f-5412-4316-83f2-fc2609f87a90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:18.265246 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223c704f-5412-4316-83f2-fc2609f87a90 None admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:18.265782 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223c704f-5412-4316-83f2-fc2609f87a90 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:18.266284 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223c704f-5412-4316-83f2-fc2609f87a90 None admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:18.267707 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-223c704f-5412-4316-83f2-fc2609f87a90 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:18.270448 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-223c704f-5412-4316-83f2-fc2609f87a90 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:18.301146 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2c63bf-a68e-41b0-a4e5-44b92162d089 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:18.302674 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2c63bf-a68e-41b0-a4e5-44b92162d089 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:18.303209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2c63bf-a68e-41b0-a4e5-44b92162d089 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:18.303693 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc2c63bf-a68e-41b0-a4e5-44b92162d089 None admin] PATH_INFO: `/v3/services/network` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:18.304654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc2c63bf-a68e-41b0-a4e5-44b92162d089 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:18.306238 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc2c63bf-a68e-41b0-a4e5-44b92162d089 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:18.310652 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:18.321661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-dc2c63bf-a68e-41b0-a4e5-44b92162d089 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Jul 12 17:20:18.321661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:18.321661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 12 17:20:18.321661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 12 17:20:18.321661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 12 17:20:18.321661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 12 17:20:18.321661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 12 17:20:18.321661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:18.321661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 12 17:20:18.321661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:18.321661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 12 17:20:18.321661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:18.321661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:18.321661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:18.321661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:18.321661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:18.321661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:18.322763 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:18.322763 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:18.322763 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:18.322763 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:18.322763 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:18.322763 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:18.322763 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 12 17:20:18.322763 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 12 17:20:18.322763 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 12 17:20:18.322763 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 12 17:20:18.322763 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:18.322763 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:18.322763 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:18.322763 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:18.322763 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:18.322763 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:18.323643 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:18.323643 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:18.323643 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:18.323643 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:18.323643 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:18.323643 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:18.323643 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:18.323643 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:18.323643 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:18.323643 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:18.323643 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 12 17:20:18.323643 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:18.323643 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 12 17:20:18.323643 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:18.325016 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 359/707] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:20:18 2022] GET /identity/v3/services/network => generated 88 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:18.325016 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2c63bf-a68e-41b0-a4e5-44b92162d089 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:18.375501 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 349/708] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Jul 12 17:20:18 2022] POST /identity/v3/users => generated 254 bytes in 281 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:20:18.377717 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223c704f-5412-4316-83f2-fc2609f87a90 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:18.542683 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc2c63bf-a68e-41b0-a4e5-44b92162d089 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:18.553889 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223c704f-5412-4316-83f2-fc2609f87a90 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:18.596658 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21f7f64f-e132-4790-8294-e23b22bd1d46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:18.598066 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f7f64f-e132-4790-8294-e23b22bd1d46 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:18.598820 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f7f64f-e132-4790-8294-e23b22bd1d46 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:18.599292 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f7f64f-e132-4790-8294-e23b22bd1d46 None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:18.601578 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21f7f64f-e132-4790-8294-e23b22bd1d46 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:18.602186 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21f7f64f-e132-4790-8294-e23b22bd1d46 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:18.604398 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21f7f64f-e132-4790-8294-e23b22bd1d46 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:18.609742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31fec146-adc2-4336-97cf-dd0bf9b429bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:18.610569 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fec146-adc2-4336-97cf-dd0bf9b429bf None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:18.610951 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fec146-adc2-4336-97cf-dd0bf9b429bf None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:18.611322 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fec146-adc2-4336-97cf-dd0bf9b429bf None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:18.612196 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31fec146-adc2-4336-97cf-dd0bf9b429bf None admin] RBAC: Adding query filter params (name=network) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:18.612602 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31fec146-adc2-4336-97cf-dd0bf9b429bf None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:18.614460 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31fec146-adc2-4336-97cf-dd0bf9b429bf None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:18.619184 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 350/709] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Jul 12 17:20:18 2022] GET /identity/v3/services?name=network => generated 124 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:18.620991 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 360/710] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 296 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:18.628001 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31fec146-adc2-4336-97cf-dd0bf9b429bf None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:18.628161 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21f7f64f-e132-4790-8294-e23b22bd1d46 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:18.638097 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:18.710973 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:18.754779 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31fec146-adc2-4336-97cf-dd0bf9b429bf None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:18.766563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21f7f64f-e132-4790-8294-e23b22bd1d46 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:18.800183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b27191f-0ec7-43aa-9ba3-2dfa8d1e6ff1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:18.800802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b27191f-0ec7-43aa-9ba3-2dfa8d1e6ff1 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:18.801241 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b27191f-0ec7-43aa-9ba3-2dfa8d1e6ff1 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:18.801531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b27191f-0ec7-43aa-9ba3-2dfa8d1e6ff1 None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:18.802568 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b27191f-0ec7-43aa-9ba3-2dfa8d1e6ff1 None admin] RBAC: Adding query filter params (type=network) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:18.802890 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b27191f-0ec7-43aa-9ba3-2dfa8d1e6ff1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:18.804533 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b27191f-0ec7-43aa-9ba3-2dfa8d1e6ff1 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:18.807459 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a70c4b-d9b8-4e91-83fc-68f5cf4d33d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:18.808520 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a70c4b-d9b8-4e91-83fc-68f5cf4d33d1 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:18.808958 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a70c4b-d9b8-4e91-83fc-68f5cf4d33d1 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:18.809390 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a70c4b-d9b8-4e91-83fc-68f5cf4d33d1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:18.810756 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 351/711] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Jul 12 17:20:18 2022] GET /identity/v3/services?type=network => generated 124 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:18.814681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa56eb9c-5bfa-4632-ab9f-7a7662acacef None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:18.815195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa56eb9c-5bfa-4632-ab9f-7a7662acacef None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:18.815640 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa56eb9c-5bfa-4632-ab9f-7a7662acacef None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:18.817546 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 352/712] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:18 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:18.824312 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25671099-8df6-41e7-a86b-6278fbb5794f None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:18.829145 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25671099-8df6-41e7-a86b-6278fbb5794f None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:18.829681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25671099-8df6-41e7-a86b-6278fbb5794f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:18.832863 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4a70c4b-d9b8-4e91-83fc-68f5cf4d33d1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:18.835432 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4a70c4b-d9b8-4e91-83fc-68f5cf4d33d1 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:18.863020 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-e4a70c4b-d9b8-4e91-83fc-68f5cf4d33d1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:18.863020 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:18.863020 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:18.863020 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:18.863020 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:18.863020 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:18.863020 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:18.863020 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:18.863020 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:18.863020 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:18.863020 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:18.863020 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:18.863020 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:20:18.863020 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:20:18.863020 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:20:18.864209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:20:18.864209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:18.864209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:18.864209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:18.864209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:18.864209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:18.864209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:18.864209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:18.864209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:18.864209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:18.864209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:18.864209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:18.864209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:18.864209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:18.864209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:18.864209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:18.864209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:18.865418 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:20:18.865418 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:20:18.865418 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:20:18.865418 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:20:18.865418 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:20:18.865418 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:20:18.865418 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:18.865418 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:18.870541 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 361/713] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:20:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:18.871862 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-25671099-8df6-41e7-a86b-6278fbb5794f None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:18.873335 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-25671099-8df6-41e7-a86b-6278fbb5794f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:18.877787 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a70c4b-d9b8-4e91-83fc-68f5cf4d33d1 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:19.015437 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:19.021466 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 353/714] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:20:19.044619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a70c4b-d9b8-4e91-83fc-68f5cf4d33d1 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:19.053202 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25671099-8df6-41e7-a86b-6278fbb5794f None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:19.072664 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:19.090888 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057718cf-895b-44fe-8464-39d6dda1d254 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:19.091935 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057718cf-895b-44fe-8464-39d6dda1d254 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:19.093013 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057718cf-895b-44fe-8464-39d6dda1d254 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:19.093685 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057718cf-895b-44fe-8464-39d6dda1d254 None admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:19.095590 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-057718cf-895b-44fe-8464-39d6dda1d254 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:19.096200 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-057718cf-895b-44fe-8464-39d6dda1d254 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:19.097848 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-057718cf-895b-44fe-8464-39d6dda1d254 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:19.113922 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 362/715] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Jul 12 17:20:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:19.123512 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057718cf-895b-44fe-8464-39d6dda1d254 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:19.186917 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25671099-8df6-41e7-a86b-6278fbb5794f None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:19.240229 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7558a5ef-283a-47bd-b66b-a5a62ef58e45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:19.241055 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7558a5ef-283a-47bd-b66b-a5a62ef58e45 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:19.241347 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7558a5ef-283a-47bd-b66b-a5a62ef58e45 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:19.241627 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7558a5ef-283a-47bd-b66b-a5a62ef58e45 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:19.242297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7558a5ef-283a-47bd-b66b-a5a62ef58e45 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:19.243709 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7558a5ef-283a-47bd-b66b-a5a62ef58e45 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:19.273224 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-7558a5ef-283a-47bd-b66b-a5a62ef58e45 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 12 17:20:19.273224 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:19.273224 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 12 17:20:19.273224 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 12 17:20:19.273224 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 12 17:20:19.273224 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 12 17:20:19.273224 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 12 17:20:19.273224 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:19.273224 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 12 17:20:19.273224 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:19.273224 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 12 17:20:19.273224 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:19.273224 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:19.273224 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:19.273224 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:19.273224 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:19.274183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:19.274183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:19.274183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:19.274183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:19.274183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:19.274183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:19.274183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:19.274183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 12 17:20:19.274183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 12 17:20:19.274183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 12 17:20:19.274183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 12 17:20:19.274183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:19.274183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:19.274183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:19.274183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:19.274183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:19.275042 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:19.275042 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:19.275042 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:19.275042 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:19.275042 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:19.275042 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:19.275042 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:19.275042 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:19.275042 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:19.275042 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:19.275042 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:19.275042 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 12 17:20:19.275042 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:19.275042 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 12 17:20:19.275042 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:19.281745 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 354/716] 10.222.0.16 () {56 vars in 1129 bytes} [Tue Jul 12 17:20:19 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:20:19.282934 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7558a5ef-283a-47bd-b66b-a5a62ef58e45 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:19.318942 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057718cf-895b-44fe-8464-39d6dda1d254 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:19.388461 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f99b2f3-b20b-4a5c-b88c-6b664dc0ad2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:19.389165 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f99b2f3-b20b-4a5c-b88c-6b664dc0ad2c None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:19.389456 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f99b2f3-b20b-4a5c-b88c-6b664dc0ad2c None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:19.389740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f99b2f3-b20b-4a5c-b88c-6b664dc0ad2c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:19.390617 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f99b2f3-b20b-4a5c-b88c-6b664dc0ad2c None admin] RBAC: Adding query filter params (role.id=eb05bf286c894ea9bf44a6c023a4323b, scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2, user.id=9012d5b88f554dc596582a7693c2a72a) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:19.390937 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f99b2f3-b20b-4a5c-b88c-6b664dc0ad2c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:19.394060 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f99b2f3-b20b-4a5c-b88c-6b664dc0ad2c None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:19.398495 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:19.414501 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 363/717] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:20:19 2022] GET /identity/v3/role_assignments?user.id=9012d5b88f554dc596582a7693c2a72a&scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2&role.id=eb05bf286c894ea9bf44a6c023a4323b => generated 259 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:19.455221 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7558a5ef-283a-47bd-b66b-a5a62ef58e45 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:19.511990 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c39f46-4b9e-40c1-a24f-76c79ce39585 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:19.512734 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c39f46-4b9e-40c1-a24f-76c79ce39585 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:19.513216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c39f46-4b9e-40c1-a24f-76c79ce39585 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:19.513506 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c39f46-4b9e-40c1-a24f-76c79ce39585 None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:19.514321 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c39f46-4b9e-40c1-a24f-76c79ce39585 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:19.514690 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c39f46-4b9e-40c1-a24f-76c79ce39585 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:19.516092 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c39f46-4b9e-40c1-a24f-76c79ce39585 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:19.521410 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 355/718] 10.222.0.16 () {56 vars in 1124 bytes} [Tue Jul 12 17:20:19 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:19.526952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f99b2f3-b20b-4a5c-b88c-6b664dc0ad2c None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:19.614833 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:19.696870 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f99b2f3-b20b-4a5c-b88c-6b664dc0ad2c None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:19.744223 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf95a11-9529-49a9-a4e1-b41401d816af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:19.744900 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf95a11-9529-49a9-a4e1-b41401d816af None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:19.745191 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf95a11-9529-49a9-a4e1-b41401d816af None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:19.745476 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf95a11-9529-49a9-a4e1-b41401d816af None admin] PATH_INFO: `/v3/services` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:19.746199 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf95a11-9529-49a9-a4e1-b41401d816af None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:19.746543 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf95a11-9529-49a9-a4e1-b41401d816af None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:19.747911 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf95a11-9529-49a9-a4e1-b41401d816af None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:19.759323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 364/719] 10.222.0.16 () {56 vars in 1124 bytes} [Tue Jul 12 17:20:19 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:19.918629 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a352ba61-c2a5-4280-892e-7a7872e160c8 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:19.918940 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a352ba61-c2a5-4280-892e-7a7872e160c8 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:19.919130 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a352ba61-c2a5-4280-892e-7a7872e160c8 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:19.920927 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 356/720] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:19.926372 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b1c277-6ba4-44ac-84c9-c03d594c9199 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:19.926711 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b1c277-6ba4-44ac-84c9-c03d594c9199 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:19.926999 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b1c277-6ba4-44ac-84c9-c03d594c9199 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:19.955655 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-36b1c277-6ba4-44ac-84c9-c03d594c9199 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:19.956908 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-36b1c277-6ba4-44ac-84c9-c03d594c9199 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:20.043438 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 365/721] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 12 17:20:20.069741 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a352ba61-c2a5-4280-892e-7a7872e160c8 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:20.215126 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a352ba61-c2a5-4280-892e-7a7872e160c8 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:20.260513 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:20.273502 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d769075-bc71-41d6-9482-ee54ecb75fa3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:20.274306 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d769075-bc71-41d6-9482-ee54ecb75fa3 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:20.274650 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d769075-bc71-41d6-9482-ee54ecb75fa3 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:20.274937 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d769075-bc71-41d6-9482-ee54ecb75fa3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:20.294052 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d769075-bc71-41d6-9482-ee54ecb75fa3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:20:20.305529 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-1d769075-bc71-41d6-9482-ee54ecb75fa3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:20:20.305529 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:20.305529 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:20.305529 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:20.305529 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:20.305529 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:20.305529 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:20.305529 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:20.305529 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:20.305529 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:20.305529 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:20.305529 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:20.305529 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:20:20.305529 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:20:20.305529 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:20:20.305529 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:20:20.306502 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:20:20.306502 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:20:20.306502 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:20:20.306502 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:20:20.306502 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:20.306502 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:20.306502 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:20.306502 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:20.306502 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:20.306502 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:20.306502 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:20.306502 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:20.306502 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:20.306502 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:20.306502 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:20.306502 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:20.307348 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:20.307348 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:20.307348 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:20.307348 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:20.307348 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:20:20.307348 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:20:20.307348 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:20:20.307348 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:20:20.307348 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:20:20.307348 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:20:20.307348 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:20:20.307348 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:20.307348 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 357/722] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Jul 12 17:20:20 2022] GET /identity/v3/roles/admin => generated 83 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:20.314000 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b1c277-6ba4-44ac-84c9-c03d594c9199 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:20.377169 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:20.450253 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b1c277-6ba4-44ac-84c9-c03d594c9199 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:20.486485 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f829089d-2b47-40c4-9ef4-137b3c1b5585 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:20.487091 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f829089d-2b47-40c4-9ef4-137b3c1b5585 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:20.487435 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f829089d-2b47-40c4-9ef4-137b3c1b5585 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:20.487722 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f829089d-2b47-40c4-9ef4-137b3c1b5585 None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:20.488479 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f829089d-2b47-40c4-9ef4-137b3c1b5585 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:20.488791 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f829089d-2b47-40c4-9ef4-137b3c1b5585 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:20.490162 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f829089d-2b47-40c4-9ef4-137b3c1b5585 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:20.500669 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 366/723] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:20:20 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:20.505962 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d769075-bc71-41d6-9482-ee54ecb75fa3 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:20.639806 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d769075-bc71-41d6-9482-ee54ecb75fa3 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:20.683012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828f77b1-7195-4bd3-9746-3a10e22db8dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:20.683812 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828f77b1-7195-4bd3-9746-3a10e22db8dc None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:20.684100 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828f77b1-7195-4bd3-9746-3a10e22db8dc None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:20.684379 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828f77b1-7195-4bd3-9746-3a10e22db8dc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:20.696396 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-828f77b1-7195-4bd3-9746-3a10e22db8dc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:20.697879 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-828f77b1-7195-4bd3-9746-3a10e22db8dc None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:20.716742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-828f77b1-7195-4bd3-9746-3a10e22db8dc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:20.716742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:20.716742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:20.716742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:20.716742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:20.716742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:20.716742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:20.716742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:20.716742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:20.716742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:20.716742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:20.716742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:20.716742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:20.716742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:20.716742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:20.718661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:20.718661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:20.718661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:20.718661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:20.718661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:20.718661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:20.718661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:20.718661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:20.718661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:20.718661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:20.718661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:20.718661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:20.718661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:20.718661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:20.718661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:20.718661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:20.718661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:20.719645 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:20.719645 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:20.719645 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:20.719645 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:20.719645 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 358/724] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:20.727828 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f829089d-2b47-40c4-9ef4-137b3c1b5585 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:20.864306 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:20.869580 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f829089d-2b47-40c4-9ef4-137b3c1b5585 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:20.922630 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be30574d-9cca-4a9b-983a-daaf1ed44f74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:20.923319 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be30574d-9cca-4a9b-983a-daaf1ed44f74 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:20.923608 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be30574d-9cca-4a9b-983a-daaf1ed44f74 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:20.923894 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be30574d-9cca-4a9b-983a-daaf1ed44f74 None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:20.924634 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be30574d-9cca-4a9b-983a-daaf1ed44f74 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:20.924952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be30574d-9cca-4a9b-983a-daaf1ed44f74 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:20.926367 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be30574d-9cca-4a9b-983a-daaf1ed44f74 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:20.941922 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 367/725] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:20.947498 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828f77b1-7195-4bd3-9746-3a10e22db8dc None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:21.024241 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:21.112653 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828f77b1-7195-4bd3-9746-3a10e22db8dc None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:21.160589 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdf9155-44f2-468c-92ca-50cbda0ae4c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:21.161310 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdf9155-44f2-468c-92ca-50cbda0ae4c7 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:21.161600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdf9155-44f2-468c-92ca-50cbda0ae4c7 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:21.161885 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdf9155-44f2-468c-92ca-50cbda0ae4c7 None admin] PATH_INFO: `/v3/users/swift` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:21.175544 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bdf9155-44f2-468c-92ca-50cbda0ae4c7 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:21.177005 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bdf9155-44f2-468c-92ca-50cbda0ae4c7 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:21.195185 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-6bdf9155-44f2-468c-92ca-50cbda0ae4c7 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 12 17:20:21.195185 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:21.195185 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:21.195185 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:21.195185 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:21.195185 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:21.195185 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:21.195185 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:21.195185 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:21.195185 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:21.195185 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:21.195185 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:21.195185 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:20:21.195185 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:20:21.195185 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:20:21.195185 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:20:21.196170 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:21.196170 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:21.196170 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:20:21.196170 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:21.196170 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:20:21.196170 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:21.196170 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:21.196170 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:21.196170 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:21.196170 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:21.196170 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:21.196170 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:21.196170 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:21.196170 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:21.196170 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:21.196170 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:21.196170 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:21.196170 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:21.197494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:21.197494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:21.197494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:20:21.197494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:20:21.197494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:20:21.197494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:20:21.197494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:20:21.197494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:20:21.197494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 12 17:20:21.197494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:21.197494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 359/726] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Jul 12 17:20:20 2022] GET /identity/v3/users/swift => generated 83 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:21.204420 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be30574d-9cca-4a9b-983a-daaf1ed44f74 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:21.237795 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa7bc9e-2898-4b6a-91fb-8de0ca05ee1a None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:21.238444 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa7bc9e-2898-4b6a-91fb-8de0ca05ee1a None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:21.238939 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa7bc9e-2898-4b6a-91fb-8de0ca05ee1a None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:21.241007 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 360/727] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 12 17:20:21.245958 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566530bc-9508-44e7-8a27-376249ec1937 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:21.246528 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566530bc-9508-44e7-8a27-376249ec1937 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:21.247025 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566530bc-9508-44e7-8a27-376249ec1937 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:21.275741 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-566530bc-9508-44e7-8a27-376249ec1937 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:21.282300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-566530bc-9508-44e7-8a27-376249ec1937 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:21.400194 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be30574d-9cca-4a9b-983a-daaf1ed44f74 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:21.414657 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 361/728] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:20:21.416321 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571693e1-eceb-4f38-a0bc-62da0c6e5e4c None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:21.416821 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571693e1-eceb-4f38-a0bc-62da0c6e5e4c None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:21.417323 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571693e1-eceb-4f38-a0bc-62da0c6e5e4c None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:21.421486 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 362/729] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:21.430933 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de0f196-2097-4960-b71a-79ee1ccff5e9 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:21.431563 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de0f196-2097-4960-b71a-79ee1ccff5e9 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:21.432053 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de0f196-2097-4960-b71a-79ee1ccff5e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:21.459027 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-8de0f196-2097-4960-b71a-79ee1ccff5e9 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:21.460389 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-8de0f196-2097-4960-b71a-79ee1ccff5e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:21.461449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1802b3-be19-4a5e-b429-e2ded8bbb75a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:21.462752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1802b3-be19-4a5e-b429-e2ded8bbb75a None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:21.463298 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1802b3-be19-4a5e-b429-e2ded8bbb75a None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:21.464822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1802b3-be19-4a5e-b429-e2ded8bbb75a None admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:21.465891 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1802b3-be19-4a5e-b429-e2ded8bbb75a None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:21.466504 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1802b3-be19-4a5e-b429-e2ded8bbb75a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:21.468549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1802b3-be19-4a5e-b429-e2ded8bbb75a None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:21.489845 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:21.497232 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 368/730] 10.222.0.16 () {56 vars in 1135 bytes} [Tue Jul 12 17:20:21 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:21.499372 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1802b3-be19-4a5e-b429-e2ded8bbb75a None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:21.527369 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:21.556852 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 363/731] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:20:21.558135 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de0f196-2097-4960-b71a-79ee1ccff5e9 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:21.647683 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1802b3-be19-4a5e-b429-e2ded8bbb75a None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:21.693672 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50cf861-d2f4-40a1-a500-0dc4b468c205 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:21.694151 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de0f196-2097-4960-b71a-79ee1ccff5e9 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:21.695129 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50cf861-d2f4-40a1-a500-0dc4b468c205 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:21.695875 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50cf861-d2f4-40a1-a500-0dc4b468c205 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:21.697236 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50cf861-d2f4-40a1-a500-0dc4b468c205 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:21.710994 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50cf861-d2f4-40a1-a500-0dc4b468c205 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:20:21.722858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-e50cf861-d2f4-40a1-a500-0dc4b468c205 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:20:21.722858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:21.722858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:21.722858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:21.722858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:21.722858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:21.722858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:21.722858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:21.722858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:21.722858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:21.722858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:21.722858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:21.722858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:20:21.722858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:20:21.722858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:20:21.722858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:20:21.723809 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:20:21.723809 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:20:21.723809 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:20:21.723809 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:20:21.723809 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:21.723809 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:21.723809 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:21.723809 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:21.723809 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:21.723809 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:21.723809 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:21.723809 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:21.723809 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:21.723809 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:21.723809 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:21.723809 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:21.724654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:21.724654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:21.724654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:21.724654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:21.724654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:20:21.724654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:20:21.724654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:20:21.724654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:20:21.724654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:20:21.724654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:20:21.724654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:20:21.724654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:21.727836 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 369/732] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Jul 12 17:20:21 2022] GET /identity/v3/roles/service => generated 85 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:21.729461 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50cf861-d2f4-40a1-a500-0dc4b468c205 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:21.735340 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42938422-a742-4f3f-9e94-df93af4f10aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:21.736110 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42938422-a742-4f3f-9e94-df93af4f10aa None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:21.736494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42938422-a742-4f3f-9e94-df93af4f10aa None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:21.736887 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42938422-a742-4f3f-9e94-df93af4f10aa None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:21.768026 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:21.813410 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42938422-a742-4f3f-9e94-df93af4f10aa None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:21.815052 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42938422-a742-4f3f-9e94-df93af4f10aa None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:21.843608 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:21.875115 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50cf861-d2f4-40a1-a500-0dc4b468c205 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:21.925354 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852c70e1-543b-495b-ab5f-2095b4a2f887 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:21.927190 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852c70e1-543b-495b-ab5f-2095b4a2f887 None admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:21.927483 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852c70e1-543b-495b-ab5f-2095b4a2f887 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:21.927774 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852c70e1-543b-495b-ab5f-2095b4a2f887 None admin] PATH_INFO: `/v3/services` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:21.928645 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852c70e1-543b-495b-ab5f-2095b4a2f887 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:21.929584 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 364/733] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:20:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 12 17:20:21.930695 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852c70e1-543b-495b-ab5f-2095b4a2f887 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:21.930806 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42938422-a742-4f3f-9e94-df93af4f10aa None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:22.033180 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 370/734] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Jul 12 17:20:21 2022] POST /identity/v3/services => generated 240 bytes in 305 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:20:22.034888 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81faa2c4-dadf-4593-be84-026c6a4d8931 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:22.035181 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81faa2c4-dadf-4593-be84-026c6a4d8931 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:22.035511 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81faa2c4-dadf-4593-be84-026c6a4d8931 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:22.036909 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 371/735] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:22.041880 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81faa2c4-dadf-4593-be84-026c6a4d8931 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:22.065942 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:22.113852 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42938422-a742-4f3f-9e94-df93af4f10aa None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:22.173292 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3012e90a-14b2-499a-b406-ff5a3be5ea0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:22.175026 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3012e90a-14b2-499a-b406-ff5a3be5ea0e None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:22.175627 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3012e90a-14b2-499a-b406-ff5a3be5ea0e None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:22.176090 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3012e90a-14b2-499a-b406-ff5a3be5ea0e None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:22.177294 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3012e90a-14b2-499a-b406-ff5a3be5ea0e None admin] RBAC: Adding query filter params (name=service) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:22.177766 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3012e90a-14b2-499a-b406-ff5a3be5ea0e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:22.180219 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3012e90a-14b2-499a-b406-ff5a3be5ea0e None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:22.193722 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 365/736] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Jul 12 17:20:21 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:22.195459 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa33c18c-a0f7-4453-8e15-bc74948ea9a1 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:22.195890 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa33c18c-a0f7-4453-8e15-bc74948ea9a1 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:22.196306 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa33c18c-a0f7-4453-8e15-bc74948ea9a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:22.208252 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81faa2c4-dadf-4593-be84-026c6a4d8931 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:22.218467 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-fa33c18c-a0f7-4453-8e15-bc74948ea9a1 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:22.220608 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-fa33c18c-a0f7-4453-8e15-bc74948ea9a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:22.256749 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6844f598-700a-4bb6-9a88-cd3b136b6842 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:22.257736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6844f598-700a-4bb6-9a88-cd3b136b6842 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:22.258101 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6844f598-700a-4bb6-9a88-cd3b136b6842 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:22.258507 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6844f598-700a-4bb6-9a88-cd3b136b6842 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:22.278214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:22.285893 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6844f598-700a-4bb6-9a88-cd3b136b6842 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:22.288327 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6844f598-700a-4bb6-9a88-cd3b136b6842 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:22.307383 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 366/737] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:22 2022] POST /identity/v3/auth/tokens => generated 898 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:20:22.308445 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa33c18c-a0f7-4453-8e15-bc74948ea9a1 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:22.313072 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-6844f598-700a-4bb6-9a88-cd3b136b6842 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:22.313072 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:22.313072 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:22.313072 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:22.313072 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:22.313072 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:22.313072 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:22.313072 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:22.313072 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:22.313072 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:22.313072 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:22.313072 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:22.313072 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:22.313072 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:22.313072 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:22.314350 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:22.314350 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:22.314350 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:22.314350 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:22.314350 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:22.314350 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:22.314350 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:22.314350 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:22.314350 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:22.314350 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:22.314350 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:22.314350 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:22.314350 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:22.314350 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:22.314350 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:22.314350 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:22.314350 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:22.315564 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:22.315564 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:22.315564 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:22.315564 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:22.317492 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 372/738] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 280 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:22.326831 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6844f598-700a-4bb6-9a88-cd3b136b6842 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:22.376428 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:22.431607 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa33c18c-a0f7-4453-8e15-bc74948ea9a1 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:22.443521 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6844f598-700a-4bb6-9a88-cd3b136b6842 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:22.474607 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0ec730-d69c-43ab-8a54-194150a92f49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:22.475796 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0ec730-d69c-43ab-8a54-194150a92f49 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:22.476287 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0ec730-d69c-43ab-8a54-194150a92f49 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:22.476799 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0ec730-d69c-43ab-8a54-194150a92f49 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:22.479447 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae44af61-89b6-4df6-90a8-1758252e4640 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:22.480117 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae44af61-89b6-4df6-90a8-1758252e4640 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:22.480405 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae44af61-89b6-4df6-90a8-1758252e4640 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:22.480689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae44af61-89b6-4df6-90a8-1758252e4640 None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:22.481415 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae44af61-89b6-4df6-90a8-1758252e4640 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:22.481723 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae44af61-89b6-4df6-90a8-1758252e4640 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:22.483384 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae44af61-89b6-4df6-90a8-1758252e4640 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:22.490572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f0ec730-d69c-43ab-8a54-194150a92f49 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:22.492397 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f0ec730-d69c-43ab-8a54-194150a92f49 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:22.498779 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 373/739] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:20:22.499696 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae44af61-89b6-4df6-90a8-1758252e4640 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:22.513350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-0f0ec730-d69c-43ab-8a54-194150a92f49 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:22.513350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:22.513350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:22.513350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:22.513350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:22.513350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:22.513350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:22.513350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:22.513350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:22.513350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:22.513350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:22.513350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:22.513350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:22.513350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:22.513350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:22.515534 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:22.515534 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:22.515534 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:22.515534 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:22.515534 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:22.515534 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:22.515534 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:22.515534 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:22.515534 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:22.515534 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:22.515534 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:22.515534 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:22.515534 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:22.515534 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:22.515534 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:22.515534 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:22.515534 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:22.516414 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:22.516414 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:22.516414 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:22.516414 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:22.517129 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 367/740] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:22.518423 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da733c1-a462-4a90-b1f5-063945ff9025 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:22.518739 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da733c1-a462-4a90-b1f5-063945ff9025 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:22.519008 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da733c1-a462-4a90-b1f5-063945ff9025 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:22.520266 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 368/741] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:22.521016 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da733c1-a462-4a90-b1f5-063945ff9025 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:22.559383 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:22.620848 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae44af61-89b6-4df6-90a8-1758252e4640 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:22.635079 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:22.640174 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da733c1-a462-4a90-b1f5-063945ff9025 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:22.659417 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a369818a-9cc5-414c-8f59-9ed4405b3a75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:22.660514 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a369818a-9cc5-414c-8f59-9ed4405b3a75 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:22.660803 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a369818a-9cc5-414c-8f59-9ed4405b3a75 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:22.661111 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a369818a-9cc5-414c-8f59-9ed4405b3a75 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:22.673985 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a369818a-9cc5-414c-8f59-9ed4405b3a75 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:20:22.679934 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a350d28-7359-4342-9156-7ac32e27b583 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:22.680719 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a350d28-7359-4342-9156-7ac32e27b583 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:22.681002 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a350d28-7359-4342-9156-7ac32e27b583 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:22.681274 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a350d28-7359-4342-9156-7ac32e27b583 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:22.681671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-a369818a-9cc5-414c-8f59-9ed4405b3a75 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:20:22.681671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:22.681671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:22.681671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:22.681671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:22.681671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:22.681671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:22.681671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:22.681671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:22.681671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:22.681671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:22.681671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:22.681671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:20:22.681671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:20:22.681671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:20:22.681671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:20:22.682595 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:20:22.682595 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:20:22.682595 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:20:22.682595 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:20:22.682595 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:22.682595 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:22.682595 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:22.682595 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:22.682595 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:22.682595 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:22.682595 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:22.682595 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:22.682595 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:22.682595 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:22.682595 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:22.682595 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:22.683430 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:22.683430 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:22.683430 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:22.683430 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:22.683430 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:20:22.683430 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:20:22.683430 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:20:22.683430 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:20:22.683430 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:20:22.683430 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:20:22.683430 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:20:22.683430 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:22.683430 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 374/742] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Jul 12 17:20:22 2022] GET /identity/v3/roles/admin => generated 83 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:22.684657 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88155df-4f85-4060-a3df-db0f59a08f80 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:22.684657 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88155df-4f85-4060-a3df-db0f59a08f80 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:22.684920 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88155df-4f85-4060-a3df-db0f59a08f80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:22.695439 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a350d28-7359-4342-9156-7ac32e27b583 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:22.696893 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a350d28-7359-4342-9156-7ac32e27b583 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:22.702697 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-f88155df-4f85-4060-a3df-db0f59a08f80 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:22.703958 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-f88155df-4f85-4060-a3df-db0f59a08f80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:22.711694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-8a350d28-7359-4342-9156-7ac32e27b583 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:22.711694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:22.711694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:22.711694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:22.711694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:22.711694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:22.711694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:22.711694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:22.711694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:22.711694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:22.711694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:22.711694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:22.711694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:20:22.711694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:20:22.711694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:20:22.712770 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:20:22.712770 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:22.712770 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:22.712770 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:22.712770 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:22.712770 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:22.712770 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:22.712770 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:22.712770 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:22.712770 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:22.712770 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:22.712770 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:22.712770 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:22.712770 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:22.712770 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:22.712770 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:22.712770 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:22.713694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:20:22.713694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:20:22.713694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:20:22.713694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:20:22.713694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:20:22.713694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:20:22.713694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:22.713694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:22.713694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 369/743] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:20:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:22.714237 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a350d28-7359-4342-9156-7ac32e27b583 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:22.777578 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 375/744] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:22 2022] POST /identity/v3/auth/tokens => generated 898 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 12 17:20:22.778776 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88155df-4f85-4060-a3df-db0f59a08f80 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:22.829558 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a350d28-7359-4342-9156-7ac32e27b583 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:22.833282 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:22.869772 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f29d397-4659-4450-80bf-fd93e12a0653 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:22.870454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f29d397-4659-4450-80bf-fd93e12a0653 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:22.870759 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f29d397-4659-4450-80bf-fd93e12a0653 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:22.871115 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f29d397-4659-4450-80bf-fd93e12a0653 None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:22.872232 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f29d397-4659-4450-80bf-fd93e12a0653 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:22.872545 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f29d397-4659-4450-80bf-fd93e12a0653 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:22.874065 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f29d397-4659-4450-80bf-fd93e12a0653 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:22.886398 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 370/745] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:22.890401 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f29d397-4659-4450-80bf-fd93e12a0653 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:22.899828 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88155df-4f85-4060-a3df-db0f59a08f80 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:22.914396 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:22.945383 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67410c6e-db15-463a-a162-3efdd129e9a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:22.946074 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67410c6e-db15-463a-a162-3efdd129e9a6 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:22.946359 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67410c6e-db15-463a-a162-3efdd129e9a6 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:22.946671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67410c6e-db15-463a-a162-3efdd129e9a6 None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:22.947438 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67410c6e-db15-463a-a162-3efdd129e9a6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:22.947751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67410c6e-db15-463a-a162-3efdd129e9a6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:22.949178 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67410c6e-db15-463a-a162-3efdd129e9a6 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:22.959968 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 376/746] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:20:22 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:22.961376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67410c6e-db15-463a-a162-3efdd129e9a6 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:22.998576 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f29d397-4659-4450-80bf-fd93e12a0653 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:23.043171 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029c2316-643b-4e7b-9e34-1804aea9f483 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:23.044281 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029c2316-643b-4e7b-9e34-1804aea9f483 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:23.044979 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029c2316-643b-4e7b-9e34-1804aea9f483 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:23.045770 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-029c2316-643b-4e7b-9e34-1804aea9f483 None admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:23.046711 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029c2316-643b-4e7b-9e34-1804aea9f483 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:23.047030 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029c2316-643b-4e7b-9e34-1804aea9f483 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:23.048473 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-029c2316-643b-4e7b-9e34-1804aea9f483 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:23.061092 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 371/747] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Jul 12 17:20:22 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:23.062010 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029c2316-643b-4e7b-9e34-1804aea9f483 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:23.096851 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67410c6e-db15-463a-a162-3efdd129e9a6 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:23.099077 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:23.137712 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc442a1-a050-4b8d-a925-06af988a2075 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:23.138391 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc442a1-a050-4b8d-a925-06af988a2075 None admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:23.139210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc442a1-a050-4b8d-a925-06af988a2075 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:23.139513 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc442a1-a050-4b8d-a925-06af988a2075 None admin] PATH_INFO: `/v3/services` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:23.140227 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc442a1-a050-4b8d-a925-06af988a2075 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:23.141587 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc442a1-a050-4b8d-a925-06af988a2075 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:23.163895 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:23.183788 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-029c2316-643b-4e7b-9e34-1804aea9f483 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:23.220077 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611c7062-eabc-453c-bbc2-f954ac0ab8fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:23.220899 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611c7062-eabc-453c-bbc2-f954ac0ab8fb None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:23.221186 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611c7062-eabc-453c-bbc2-f954ac0ab8fb None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:23.221467 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611c7062-eabc-453c-bbc2-f954ac0ab8fb None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:23.230029 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-611c7062-eabc-453c-bbc2-f954ac0ab8fb None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:23.231508 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-611c7062-eabc-453c-bbc2-f954ac0ab8fb None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:23.241666 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 377/748] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Jul 12 17:20:22 2022] POST /identity/v3/services => generated 251 bytes in 281 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:20:23.246465 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc442a1-a050-4b8d-a925-06af988a2075 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:23.247128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-611c7062-eabc-453c-bbc2-f954ac0ab8fb None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 12 17:20:23.247128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:23.247128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:23.247128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:23.247128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:23.247128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:23.247128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:23.247128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:23.247128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:23.247128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:23.247128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:23.247128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:23.247128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:20:23.247128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:20:23.247128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:20:23.247128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:20:23.248849 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:23.248849 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:23.248849 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:20:23.248849 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:23.248849 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:20:23.248849 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:23.248849 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:23.248849 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:23.248849 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:23.248849 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:23.248849 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:23.248849 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:23.248849 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:23.248849 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:23.248849 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:23.248849 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:23.248849 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:23.248849 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:23.250056 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:23.250056 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:23.250056 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:20:23.250056 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:20:23.250056 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:20:23.250056 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:20:23.250056 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:20:23.250056 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:20:23.250056 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 12 17:20:23.250056 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:23.252771 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 372/749] 10.222.0.16 () {56 vars in 1117 bytes} [Tue Jul 12 17:20:23 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:23.253887 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611c7062-eabc-453c-bbc2-f954ac0ab8fb None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:23.382537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611c7062-eabc-453c-bbc2-f954ac0ab8fb None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:23.415866 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc442a1-a050-4b8d-a925-06af988a2075 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:23.425054 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718a7acd-57b2-442d-a681-7f8cad7a1d11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:23.425722 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718a7acd-57b2-442d-a681-7f8cad7a1d11 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:23.426012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718a7acd-57b2-442d-a681-7f8cad7a1d11 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:23.426295 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718a7acd-57b2-442d-a681-7f8cad7a1d11 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:23.427173 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718a7acd-57b2-442d-a681-7f8cad7a1d11 None admin] RBAC: Adding query filter params (role.id=eb05bf286c894ea9bf44a6c023a4323b, scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2, user.id=d7906d1a19c642bd93507a27bf3e3dae) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:23.427495 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718a7acd-57b2-442d-a681-7f8cad7a1d11 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:23.428950 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718a7acd-57b2-442d-a681-7f8cad7a1d11 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:23.435168 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 373/750] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:20:23 2022] GET /identity/v3/role_assignments?user.id=d7906d1a19c642bd93507a27bf3e3dae&scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2&role.id=eb05bf286c894ea9bf44a6c023a4323b => generated 259 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:23.436529 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718a7acd-57b2-442d-a681-7f8cad7a1d11 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:23.440053 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:23.464547 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11269fb9-0956-4df1-adb4-78def518e5fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:23.466074 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11269fb9-0956-4df1-adb4-78def518e5fc None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:23.466898 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11269fb9-0956-4df1-adb4-78def518e5fc None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:23.468950 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11269fb9-0956-4df1-adb4-78def518e5fc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:23.484868 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11269fb9-0956-4df1-adb4-78def518e5fc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:23.488161 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11269fb9-0956-4df1-adb4-78def518e5fc None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:23.515324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-11269fb9-0956-4df1-adb4-78def518e5fc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:23.515324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:23.515324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:23.515324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:23.515324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:23.515324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:23.515324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:23.515324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:23.515324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:23.515324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:23.515324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:23.515324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:23.515324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:23.515324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:23.515324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:23.516323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:23.516323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:23.516323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:23.516323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:23.516323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:23.516323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:23.516323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:23.516323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:23.516323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:23.516323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:23.516323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:23.516323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:23.516323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:23.516323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:23.516323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:23.516323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:23.516323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:23.517219 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:23.517219 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:23.517219 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:23.517219 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:23.517219 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 378/751] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 275 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:23.522584 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11269fb9-0956-4df1-adb4-78def518e5fc None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:23.538288 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:23.575865 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718a7acd-57b2-442d-a681-7f8cad7a1d11 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:23.615527 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab69169b-cc7f-40ea-9308-bc6809ef2f72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:23.616219 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab69169b-cc7f-40ea-9308-bc6809ef2f72 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:23.616508 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab69169b-cc7f-40ea-9308-bc6809ef2f72 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:23.616789 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab69169b-cc7f-40ea-9308-bc6809ef2f72 None admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:23.617658 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab69169b-cc7f-40ea-9308-bc6809ef2f72 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:23.617977 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab69169b-cc7f-40ea-9308-bc6809ef2f72 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:23.619501 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab69169b-cc7f-40ea-9308-bc6809ef2f72 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:23.640446 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 374/752] 10.222.0.16 () {56 vars in 1149 bytes} [Tue Jul 12 17:20:23 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:23.646206 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab69169b-cc7f-40ea-9308-bc6809ef2f72 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:23.648311 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11269fb9-0956-4df1-adb4-78def518e5fc None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:23.687943 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fbe7be4-ca5e-4a3f-ae96-ce5d19c99943 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:23.688997 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fbe7be4-ca5e-4a3f-ae96-ce5d19c99943 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:23.689505 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fbe7be4-ca5e-4a3f-ae96-ce5d19c99943 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:23.689982 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fbe7be4-ca5e-4a3f-ae96-ce5d19c99943 None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:23.690959 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fbe7be4-ca5e-4a3f-ae96-ce5d19c99943 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:23.691608 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fbe7be4-ca5e-4a3f-ae96-ce5d19c99943 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:23.693202 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fbe7be4-ca5e-4a3f-ae96-ce5d19c99943 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:23.708017 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 379/753] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:20:23.713910 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fbe7be4-ca5e-4a3f-ae96-ce5d19c99943 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:23.790860 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:23.843280 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab69169b-cc7f-40ea-9308-bc6809ef2f72 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:23.869287 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:23.915626 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fbe7be4-ca5e-4a3f-ae96-ce5d19c99943 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:23.919748 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f92e2d25-a18f-4012-9f63-54439e781532 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:23.921522 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92e2d25-a18f-4012-9f63-54439e781532 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:23.921810 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92e2d25-a18f-4012-9f63-54439e781532 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:23.922090 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92e2d25-a18f-4012-9f63-54439e781532 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:23.956066 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995e0ef0-8763-4d6e-bab4-3728ae6b6038 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:23.957273 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995e0ef0-8763-4d6e-bab4-3728ae6b6038 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:23.957763 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995e0ef0-8763-4d6e-bab4-3728ae6b6038 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:23.958279 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995e0ef0-8763-4d6e-bab4-3728ae6b6038 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:23.967553 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995e0ef0-8763-4d6e-bab4-3728ae6b6038 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:23.969423 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995e0ef0-8763-4d6e-bab4-3728ae6b6038 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:23.986617 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-995e0ef0-8763-4d6e-bab4-3728ae6b6038 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 12 17:20:23.986617 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:23.986617 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:23.986617 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:23.986617 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:23.986617 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:23.986617 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:23.986617 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:23.986617 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:23.986617 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:23.986617 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:23.986617 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:23.986617 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:20:23.986617 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:20:23.986617 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:20:23.986617 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:20:23.987623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:23.987623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:23.987623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:20:23.987623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:23.987623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:20:23.987623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:23.987623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:23.987623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:23.987623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:23.987623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:23.987623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:23.987623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:23.987623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:23.987623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:23.987623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:23.987623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:23.987623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:23.987623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:23.988559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:23.988559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:23.988559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:20:23.988559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:20:23.988559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:20:23.988559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:20:23.988559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:20:23.988559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:20:23.988559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 12 17:20:23.988559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:23.991441 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 380/754] 10.222.0.16 () {56 vars in 1093 bytes} [Tue Jul 12 17:20:23 2022] GET /identity/v3/users/nova => generated 82 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:24.000745 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995e0ef0-8763-4d6e-bab4-3728ae6b6038 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:24.015827 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f92e2d25-a18f-4012-9f63-54439e781532 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:24.017326 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f92e2d25-a18f-4012-9f63-54439e781532 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:24.114204 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995e0ef0-8763-4d6e-bab4-3728ae6b6038 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:24.145561 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:24.152531 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c08e8a-c8a3-400c-8346-8d78a59a11f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:24.153540 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c08e8a-c8a3-400c-8346-8d78a59a11f7 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:24.154030 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c08e8a-c8a3-400c-8346-8d78a59a11f7 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:24.155111 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4c08e8a-c8a3-400c-8346-8d78a59a11f7 None admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:24.156515 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c08e8a-c8a3-400c-8346-8d78a59a11f7 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:24.157057 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c08e8a-c8a3-400c-8346-8d78a59a11f7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:24.158731 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4c08e8a-c8a3-400c-8346-8d78a59a11f7 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:24.169994 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:24.180544 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 381/755] 10.222.0.16 () {56 vars in 1133 bytes} [Tue Jul 12 17:20:23 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:24.187631 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c08e8a-c8a3-400c-8346-8d78a59a11f7 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:24.230407 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 375/756] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:20:23 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 580 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 12 17:20:24.231773 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f92e2d25-a18f-4012-9f63-54439e781532 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:24.324665 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4c08e8a-c8a3-400c-8346-8d78a59a11f7 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:24.361279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f92e2d25-a18f-4012-9f63-54439e781532 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:24.365244 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e26752-b457-49df-82e5-6d6e1d897972 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:24.366040 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e26752-b457-49df-82e5-6d6e1d897972 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:24.366329 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e26752-b457-49df-82e5-6d6e1d897972 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:24.366634 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e26752-b457-49df-82e5-6d6e1d897972 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:24.378756 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e26752-b457-49df-82e5-6d6e1d897972 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:24.380432 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e26752-b457-49df-82e5-6d6e1d897972 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:24.398397 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03409e3-e61e-4c22-9305-9b4ae3abc82d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:24.399382 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03409e3-e61e-4c22-9305-9b4ae3abc82d None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:24.399610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-e0e26752-b457-49df-82e5-6d6e1d897972 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:24.399610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:24.399610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:24.399610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:24.399610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:24.399610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:24.399610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:24.399610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:24.399610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:24.399610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:24.399610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:24.399610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:24.399610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:24.399610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:24.399610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:24.400488 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:24.400488 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:24.400488 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:24.400488 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:24.400488 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:24.400488 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:24.400488 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:24.400488 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:24.400488 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:24.400488 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:24.400488 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:24.400488 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:24.400488 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:24.400488 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:24.400488 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:24.400488 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:24.400488 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:24.401390 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:24.401390 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:24.401390 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:24.401390 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:24.401620 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03409e3-e61e-4c22-9305-9b4ae3abc82d None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:24.401620 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03409e3-e61e-4c22-9305-9b4ae3abc82d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:24.401746 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 382/757] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:24.402465 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7312e1bf-d8c0-46b0-bd2a-1767feaac523 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:24.402935 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7312e1bf-d8c0-46b0-bd2a-1767feaac523 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:24.406455 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7312e1bf-d8c0-46b0-bd2a-1767feaac523 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:24.409147 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 383/758] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:24.409951 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7312e1bf-d8c0-46b0-bd2a-1767feaac523 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:24.416371 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d03409e3-e61e-4c22-9305-9b4ae3abc82d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:24.417830 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d03409e3-e61e-4c22-9305-9b4ae3abc82d None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:24.436265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-d03409e3-e61e-4c22-9305-9b4ae3abc82d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:24.436265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:24.436265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:24.436265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:24.436265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:24.436265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:24.436265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:24.436265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:24.436265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:24.436265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:24.436265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:24.436265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:24.436265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:24.436265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:24.436265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:24.438142 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:24.438142 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:24.438142 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:24.438142 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:24.438142 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:24.438142 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:24.438142 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:24.438142 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:24.438142 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:24.438142 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:24.438142 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:24.438142 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:24.438142 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:24.438142 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:24.438142 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:24.438142 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:24.438142 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:24.440342 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:24.440342 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:24.440342 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:24.440342 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:24.440342 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 376/759] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:24.440342 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c271c1b9-9584-48ef-a3fd-9fdc0839270d None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:24.440342 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c271c1b9-9584-48ef-a3fd-9fdc0839270d None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:24.440342 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c271c1b9-9584-48ef-a3fd-9fdc0839270d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:24.457034 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-c271c1b9-9584-48ef-a3fd-9fdc0839270d None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:24.458149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:24.458926 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-c271c1b9-9584-48ef-a3fd-9fdc0839270d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:24.469432 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:24.524868 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7312e1bf-d8c0-46b0-bd2a-1767feaac523 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:24.534461 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 377/760] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:24 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:20:24.534461 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c271c1b9-9584-48ef-a3fd-9fdc0839270d None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:24.600716 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15fc8c43-739b-4b53-b04c-c2490967c7fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:24.601643 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15fc8c43-739b-4b53-b04c-c2490967c7fe None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:24.602089 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15fc8c43-739b-4b53-b04c-c2490967c7fe None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:24.602785 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15fc8c43-739b-4b53-b04c-c2490967c7fe None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:24.603749 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15fc8c43-739b-4b53-b04c-c2490967c7fe None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:24.604200 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15fc8c43-739b-4b53-b04c-c2490967c7fe None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:24.605778 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15fc8c43-739b-4b53-b04c-c2490967c7fe None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:24.618841 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 384/761] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:24.619858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15fc8c43-739b-4b53-b04c-c2490967c7fe None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:24.653177 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c271c1b9-9584-48ef-a3fd-9fdc0839270d None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:24.691683 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003ade2e-52ab-447b-8a96-16efa8a9f8c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:24.692371 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003ade2e-52ab-447b-8a96-16efa8a9f8c7 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:24.692659 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003ade2e-52ab-447b-8a96-16efa8a9f8c7 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:24.692961 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003ade2e-52ab-447b-8a96-16efa8a9f8c7 None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:24.693960 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003ade2e-52ab-447b-8a96-16efa8a9f8c7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:24.694822 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003ade2e-52ab-447b-8a96-16efa8a9f8c7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:24.696591 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003ade2e-52ab-447b-8a96-16efa8a9f8c7 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:24.711722 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 378/762] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:24.712605 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003ade2e-52ab-447b-8a96-16efa8a9f8c7 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:24.717947 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:24.735473 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:24.739330 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15fc8c43-739b-4b53-b04c-c2490967c7fe None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:24.784936 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0d3bdc-de43-45f5-babc-4fa4c6a4b466 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:24.786039 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0d3bdc-de43-45f5-babc-4fa4c6a4b466 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:24.786339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0d3bdc-de43-45f5-babc-4fa4c6a4b466 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:24.786655 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0d3bdc-de43-45f5-babc-4fa4c6a4b466 None admin] PATH_INFO: `/v3/services/network` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:24.787328 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0d3bdc-de43-45f5-babc-4fa4c6a4b466 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:24.789215 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0d3bdc-de43-45f5-babc-4fa4c6a4b466 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:24.805842 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-4a0d3bdc-de43-45f5-babc-4fa4c6a4b466 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Jul 12 17:20:24.805842 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:24.805842 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 12 17:20:24.805842 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 12 17:20:24.805842 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 12 17:20:24.805842 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 12 17:20:24.805842 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 12 17:20:24.805842 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:24.805842 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 12 17:20:24.805842 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:24.805842 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 12 17:20:24.805842 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:24.805842 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:24.805842 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:24.805842 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:24.805842 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:24.805842 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:24.806888 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:24.806888 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:24.806888 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:24.806888 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:24.806888 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:24.806888 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:24.806888 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 12 17:20:24.806888 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 12 17:20:24.806888 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 12 17:20:24.806888 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 12 17:20:24.806888 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:24.806888 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:24.806888 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:24.806888 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:24.806888 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:24.806888 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:24.807793 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:24.807793 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:24.807793 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:24.807793 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:24.807793 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:24.807793 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:24.807793 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:24.807793 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:24.807793 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:24.807793 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:24.807793 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 12 17:20:24.807793 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:24.807793 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 12 17:20:24.807793 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:24.812801 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 385/763] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:20:24 2022] GET /identity/v3/services/network => generated 88 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:24.814297 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0d3bdc-de43-45f5-babc-4fa4c6a4b466 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:24.842872 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003ade2e-52ab-447b-8a96-16efa8a9f8c7 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:24.885111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4672c597-52d4-4761-b1d9-51960c66d501 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:24.886181 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4672c597-52d4-4761-b1d9-51960c66d501 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:24.886509 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4672c597-52d4-4761-b1d9-51960c66d501 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:24.886797 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4672c597-52d4-4761-b1d9-51960c66d501 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:24.896219 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4672c597-52d4-4761-b1d9-51960c66d501 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:24.898169 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4672c597-52d4-4761-b1d9-51960c66d501 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:24.916275 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-4672c597-52d4-4761-b1d9-51960c66d501 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:24.916275 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:24.916275 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:24.916275 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:24.916275 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:24.916275 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:24.916275 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:24.916275 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:24.916275 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:24.916275 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:24.916275 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:24.916275 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:24.916275 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:20:24.916275 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:20:24.916275 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:20:24.917200 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:20:24.917200 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:24.917200 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:24.917200 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:24.917200 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:24.917200 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:24.917200 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:24.917200 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:24.917200 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:24.917200 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:24.917200 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:24.917200 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:24.917200 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:24.917200 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:24.917200 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:24.917200 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:24.917200 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:24.918079 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:20:24.918079 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:20:24.918079 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:20:24.918079 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:20:24.918079 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:20:24.918079 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:20:24.918079 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:24.918079 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:24.918079 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 379/764] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:20:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:24.919049 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4672c597-52d4-4761-b1d9-51960c66d501 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:24.939562 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0d3bdc-de43-45f5-babc-4fa4c6a4b466 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:24.978309 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3a87d6-8b2d-4b7b-b64a-5a58526df762 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:24.979647 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3a87d6-8b2d-4b7b-b64a-5a58526df762 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:24.980133 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3a87d6-8b2d-4b7b-b64a-5a58526df762 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:24.980655 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3a87d6-8b2d-4b7b-b64a-5a58526df762 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:24.988668 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f3a87d6-8b2d-4b7b-b64a-5a58526df762 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:24.990516 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f3a87d6-8b2d-4b7b-b64a-5a58526df762 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:24.997387 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:25.011749 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-5f3a87d6-8b2d-4b7b-b64a-5a58526df762 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:25.011749 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:25.011749 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:25.011749 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:25.011749 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:25.011749 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:25.011749 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:25.011749 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:25.011749 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:25.011749 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:25.011749 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:25.011749 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:25.011749 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:20:25.011749 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:20:25.011749 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:20:25.012657 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:20:25.012657 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:25.012657 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:25.012657 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:25.012657 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:25.012657 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:25.012657 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:25.012657 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:25.012657 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:25.012657 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:25.012657 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:25.012657 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:25.012657 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:25.012657 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:25.012657 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:25.012657 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:25.012657 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:25.014209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:20:25.014209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:20:25.014209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:20:25.014209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:20:25.014209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:20:25.014209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:20:25.014209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:25.014209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:25.021309 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 386/765] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:20:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:25.021309 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3a87d6-8b2d-4b7b-b64a-5a58526df762 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:25.047893 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:25.051202 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4672c597-52d4-4761-b1d9-51960c66d501 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:25.089872 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c2ee6b-a9f7-46bb-be9c-caf7df92a805 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:25.091043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c2ee6b-a9f7-46bb-be9c-caf7df92a805 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:25.091531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c2ee6b-a9f7-46bb-be9c-caf7df92a805 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:25.092091 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c2ee6b-a9f7-46bb-be9c-caf7df92a805 None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:25.093078 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c2ee6b-a9f7-46bb-be9c-caf7df92a805 None admin] RBAC: Adding query filter params (name=network) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:25.093732 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c2ee6b-a9f7-46bb-be9c-caf7df92a805 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:25.095557 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c2ee6b-a9f7-46bb-be9c-caf7df92a805 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:25.101097 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 380/766] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Jul 12 17:20:24 2022] GET /identity/v3/services?name=network => generated 124 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:25.102532 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c2ee6b-a9f7-46bb-be9c-caf7df92a805 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:25.133100 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3a87d6-8b2d-4b7b-b64a-5a58526df762 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:25.169512 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4567b0-04f4-4ed3-9e36-eae0a0fc1a03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:25.170480 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4567b0-04f4-4ed3-9e36-eae0a0fc1a03 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:25.170807 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4567b0-04f4-4ed3-9e36-eae0a0fc1a03 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:25.171090 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4567b0-04f4-4ed3-9e36-eae0a0fc1a03 None admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:25.171895 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa4567b0-04f4-4ed3-9e36-eae0a0fc1a03 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:25.172205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa4567b0-04f4-4ed3-9e36-eae0a0fc1a03 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:25.173643 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa4567b0-04f4-4ed3-9e36-eae0a0fc1a03 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:25.186867 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 387/767] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Jul 12 17:20:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:25.187820 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4567b0-04f4-4ed3-9e36-eae0a0fc1a03 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:25.218389 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c2ee6b-a9f7-46bb-be9c-caf7df92a805 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:25.250208 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:25.259747 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89901f6b-40a2-452d-a3be-01143f35017d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:25.264353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89901f6b-40a2-452d-a3be-01143f35017d None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:25.264353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89901f6b-40a2-452d-a3be-01143f35017d None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:25.264353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89901f6b-40a2-452d-a3be-01143f35017d None admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:25.264353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89901f6b-40a2-452d-a3be-01143f35017d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:25.264353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89901f6b-40a2-452d-a3be-01143f35017d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:25.265162 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89901f6b-40a2-452d-a3be-01143f35017d None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:25.283802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 381/768] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Jul 12 17:20:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:25.284561 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:25.286141 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89901f6b-40a2-452d-a3be-01143f35017d None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:25.297284 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4567b0-04f4-4ed3-9e36-eae0a0fc1a03 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:25.333893 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53188068-f2c0-41e8-8ea0-afe3ff97c713 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:25.334664 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53188068-f2c0-41e8-8ea0-afe3ff97c713 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:25.334891 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53188068-f2c0-41e8-8ea0-afe3ff97c713 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:25.335179 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53188068-f2c0-41e8-8ea0-afe3ff97c713 None admin] PATH_INFO: `/v3/services` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:25.335894 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53188068-f2c0-41e8-8ea0-afe3ff97c713 None admin] RBAC: Adding query filter params (type=network) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:25.336212 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53188068-f2c0-41e8-8ea0-afe3ff97c713 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:25.337641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53188068-f2c0-41e8-8ea0-afe3ff97c713 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:25.348480 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 388/769] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Jul 12 17:20:25 2022] GET /identity/v3/services?type=network => generated 350 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:25.349300 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53188068-f2c0-41e8-8ea0-afe3ff97c713 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:25.405202 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89901f6b-40a2-452d-a3be-01143f35017d None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:25.443056 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47c4ac8-4ed9-4ba0-ae26-ad80274e1ea0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:25.443958 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47c4ac8-4ed9-4ba0-ae26-ad80274e1ea0 None admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:25.444248 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47c4ac8-4ed9-4ba0-ae26-ad80274e1ea0 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:25.444531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47c4ac8-4ed9-4ba0-ae26-ad80274e1ea0 None admin] PATH_INFO: `/v3/projects/fd4ed2fe037c4fb7ba49c724eb9414c2/users/9012d5b88f554dc596582a7693c2a72a/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:25.446668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a47c4ac8-4ed9-4ba0-ae26-ad80274e1ea0 None admin] RBAC: Authorizing `identity:create_grant(project_id=fd4ed2fe037c4fb7ba49c724eb9414c2, user_id=9012d5b88f554dc596582a7693c2a72a, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:25.448249 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a47c4ac8-4ed9-4ba0-ae26-ad80274e1ea0 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:25.473526 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53188068-f2c0-41e8-8ea0-afe3ff97c713 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:25.510870 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ab3554-7878-45fc-ae14-de04e231ba11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:25.511951 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ab3554-7878-45fc-ae14-de04e231ba11 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:25.512452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ab3554-7878-45fc-ae14-de04e231ba11 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:25.512935 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ab3554-7878-45fc-ae14-de04e231ba11 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:25.514164 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ab3554-7878-45fc-ae14-de04e231ba11 None admin] RBAC: Adding query filter params (role.id=bef74d4975164f75bcec96ac47afea73, scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2, user.id=3ac0d5d8de4943a3888abee3b24b8a6f) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:25.514827 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ab3554-7878-45fc-ae14-de04e231ba11 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:25.516566 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ab3554-7878-45fc-ae14-de04e231ba11 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:25.519163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 382/770] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Jul 12 17:20:25 2022] PUT /identity/v3/projects/fd4ed2fe037c4fb7ba49c724eb9414c2/users/9012d5b88f554dc596582a7693c2a72a/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:20:25.520208 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47c4ac8-4ed9-4ba0-ae26-ad80274e1ea0 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:25.530925 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 389/771] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:20:25 2022] GET /identity/v3/role_assignments?user.id=3ac0d5d8de4943a3888abee3b24b8a6f&scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2&role.id=bef74d4975164f75bcec96ac47afea73 => generated 259 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:25.600956 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:25.648591 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47c4ac8-4ed9-4ba0-ae26-ad80274e1ea0 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:25.690191 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3cd1ef1-d2bd-4d1b-bf9a-e8cb47b68f89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:25.690922 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3cd1ef1-d2bd-4d1b-bf9a-e8cb47b68f89 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:25.691215 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3cd1ef1-d2bd-4d1b-bf9a-e8cb47b68f89 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:25.691500 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3cd1ef1-d2bd-4d1b-bf9a-e8cb47b68f89 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:25.692337 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3cd1ef1-d2bd-4d1b-bf9a-e8cb47b68f89 None admin] RBAC: Adding query filter params (interface=public, service_id=ed28c31ded3f42088bd27ea38094fd88, region_id=RegionOne) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:25.692662 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3cd1ef1-d2bd-4d1b-bf9a-e8cb47b68f89 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:25.693157 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58e6ef1-6806-4ca3-8f66-b8ba8e5b5bd4 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:25.693665 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58e6ef1-6806-4ca3-8f66-b8ba8e5b5bd4 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:25.694156 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58e6ef1-6806-4ca3-8f66-b8ba8e5b5bd4 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:25.694261 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3cd1ef1-d2bd-4d1b-bf9a-e8cb47b68f89 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:25.696292 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 390/772] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:25 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 12 17:20:25.702551 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f38ad7-bbc2-47d4-a978-8ab748038ce8 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:25.703199 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f38ad7-bbc2-47d4-a978-8ab748038ce8 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:25.703695 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f38ad7-bbc2-47d4-a978-8ab748038ce8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:25.708761 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 383/773] 10.222.0.16 () {56 vars in 1251 bytes} [Tue Jul 12 17:20:25 2022] GET /identity/v3/endpoints?service_id=ed28c31ded3f42088bd27ea38094fd88&interface=public®ion_id=RegionOne => generated 194 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:25.710273 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3cd1ef1-d2bd-4d1b-bf9a-e8cb47b68f89 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:25.738913 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-82f38ad7-bbc2-47d4-a978-8ab748038ce8 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:25.740506 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-82f38ad7-bbc2-47d4-a978-8ab748038ce8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:25.753524 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:25.828596 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 391/774] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:25 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:20:25.848690 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3cd1ef1-d2bd-4d1b-bf9a-e8cb47b68f89 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:25.858884 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f38ad7-bbc2-47d4-a978-8ab748038ce8 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:25.864461 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:25.891887 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b57bee-2125-4da5-a006-8b6966bc1ad1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:25.892749 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b57bee-2125-4da5-a006-8b6966bc1ad1 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:25.893476 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b57bee-2125-4da5-a006-8b6966bc1ad1 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:25.893765 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b57bee-2125-4da5-a006-8b6966bc1ad1 None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:25.894772 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b57bee-2125-4da5-a006-8b6966bc1ad1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:25.896391 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b57bee-2125-4da5-a006-8b6966bc1ad1 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:25.909251 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 384/775] 10.222.0.16 () {56 vars in 1087 bytes} [Tue Jul 12 17:20:25 2022] GET /identity/v3/services => generated 772 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:20:25.910793 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aceb6cbb-b383-4c4b-a2f3-4c80f6b4a015 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:25.911093 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aceb6cbb-b383-4c4b-a2f3-4c80f6b4a015 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:25.911379 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aceb6cbb-b383-4c4b-a2f3-4c80f6b4a015 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:25.913358 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 385/776] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:25.921416 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b9dee9-4f92-4ac5-aad0-55852630f606 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:25.921740 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b9dee9-4f92-4ac5-aad0-55852630f606 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:25.921990 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b9dee9-4f92-4ac5-aad0-55852630f606 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:25.963632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-f7b9dee9-4f92-4ac5-aad0-55852630f606 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:25.965254 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-f7b9dee9-4f92-4ac5-aad0-55852630f606 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:26.014167 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f38ad7-bbc2-47d4-a978-8ab748038ce8 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:26.078724 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18631ed-48ff-43c4-b0b7-a55fa27b9840 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:26.079466 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18631ed-48ff-43c4-b0b7-a55fa27b9840 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:26.079756 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18631ed-48ff-43c4-b0b7-a55fa27b9840 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:26.080040 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18631ed-48ff-43c4-b0b7-a55fa27b9840 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:26.082691 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 386/777] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:25 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 12 17:20:26.091472 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e18631ed-48ff-43c4-b0b7-a55fa27b9840 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:20:26.099479 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-e18631ed-48ff-43c4-b0b7-a55fa27b9840 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:20:26.099479 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:26.099479 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:26.099479 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:26.099479 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:26.099479 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:26.099479 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:26.099479 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:26.099479 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:26.099479 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:26.099479 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:26.099479 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:26.099479 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:20:26.099479 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:20:26.099479 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:20:26.099479 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:20:26.100467 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:20:26.100467 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:20:26.100467 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:20:26.100467 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:20:26.100467 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:26.100467 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:26.100467 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:26.100467 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:26.100467 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:26.100467 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:26.100467 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:26.100467 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:26.100467 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:26.100467 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:26.100467 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:26.100467 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:26.101305 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:26.101305 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:26.101305 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:26.101305 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:26.101305 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:20:26.101305 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:20:26.101305 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:20:26.101305 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:20:26.101305 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:20:26.101305 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:20:26.101305 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:20:26.101305 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:26.105173 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 392/778] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Jul 12 17:20:25 2022] GET /identity/v3/roles/admin => generated 83 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:20:26.106394 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b9dee9-4f92-4ac5-aad0-55852630f606 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:26.107936 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18631ed-48ff-43c4-b0b7-a55fa27b9840 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:26.112327 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:26.233066 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b9dee9-4f92-4ac5-aad0-55852630f606 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:26.235647 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18631ed-48ff-43c4-b0b7-a55fa27b9840 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:26.241324 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:26.272236 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b491cbf4-b5c2-4c0f-9bfa-487d74ae9ff0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:26.272916 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b491cbf4-b5c2-4c0f-9bfa-487d74ae9ff0 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:26.273204 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b491cbf4-b5c2-4c0f-9bfa-487d74ae9ff0 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:26.273486 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b491cbf4-b5c2-4c0f-9bfa-487d74ae9ff0 None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:26.274247 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b491cbf4-b5c2-4c0f-9bfa-487d74ae9ff0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:26.274595 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b491cbf4-b5c2-4c0f-9bfa-487d74ae9ff0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:26.274700 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af248f7-bdb4-4cc6-b4e8-f750810e8fc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:26.275484 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af248f7-bdb4-4cc6-b4e8-f750810e8fc1 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:26.275776 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af248f7-bdb4-4cc6-b4e8-f750810e8fc1 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:26.275980 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b491cbf4-b5c2-4c0f-9bfa-487d74ae9ff0 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:26.276086 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af248f7-bdb4-4cc6-b4e8-f750810e8fc1 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:26.276707 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af248f7-bdb4-4cc6-b4e8-f750810e8fc1 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:26.278274 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af248f7-bdb4-4cc6-b4e8-f750810e8fc1 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:26.285475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 393/779] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:20:26 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:26.293298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-3af248f7-bdb4-4cc6-b4e8-f750810e8fc1 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 12 17:20:26.293298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:26.293298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 12 17:20:26.293298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 12 17:20:26.293298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 12 17:20:26.293298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 12 17:20:26.293298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 12 17:20:26.293298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:26.293298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 12 17:20:26.293298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:26.293298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 12 17:20:26.293298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:26.293298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:26.293298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:26.293298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:26.293298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:26.294324 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:26.294324 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:26.294324 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:26.294324 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:26.294324 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:26.294324 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:26.294324 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:26.294324 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 12 17:20:26.294324 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 12 17:20:26.294324 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 12 17:20:26.294324 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 12 17:20:26.294324 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:26.294324 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:26.294324 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:26.294324 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:26.294324 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:26.295279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:26.295279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:26.295279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:26.295279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:26.295279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:26.295279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:26.295279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:26.295279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:26.295279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:26.295279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:26.295279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:26.295279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 12 17:20:26.295279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:26.295279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 12 17:20:26.295279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:26.295279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 387/780] 10.222.0.16 () {56 vars in 1129 bytes} [Tue Jul 12 17:20:26 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:26.296123 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b491cbf4-b5c2-4c0f-9bfa-487d74ae9ff0 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:26.300990 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af248f7-bdb4-4cc6-b4e8-f750810e8fc1 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:26.408483 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:26.415517 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af248f7-bdb4-4cc6-b4e8-f750810e8fc1 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:26.451633 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b491cbf4-b5c2-4c0f-9bfa-487d74ae9ff0 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:26.452767 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90fe1e7-0351-42d9-9a0c-de1d969e9209 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:26.453612 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90fe1e7-0351-42d9-9a0c-de1d969e9209 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:26.454073 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90fe1e7-0351-42d9-9a0c-de1d969e9209 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:26.454523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90fe1e7-0351-42d9-9a0c-de1d969e9209 None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:26.455392 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90fe1e7-0351-42d9-9a0c-de1d969e9209 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:26.455712 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90fe1e7-0351-42d9-9a0c-de1d969e9209 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:26.457359 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d90fe1e7-0351-42d9-9a0c-de1d969e9209 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:26.473604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 388/781] 10.222.0.16 () {56 vars in 1124 bytes} [Tue Jul 12 17:20:26 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:20:26.479668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90fe1e7-0351-42d9-9a0c-de1d969e9209 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:26.491931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24933f59-ccf7-4fbe-a321-8dc19edb3a32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:26.493180 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24933f59-ccf7-4fbe-a321-8dc19edb3a32 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:26.493726 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24933f59-ccf7-4fbe-a321-8dc19edb3a32 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:26.494762 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24933f59-ccf7-4fbe-a321-8dc19edb3a32 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:26.511426 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24933f59-ccf7-4fbe-a321-8dc19edb3a32 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:26.513256 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24933f59-ccf7-4fbe-a321-8dc19edb3a32 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:26.535984 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-24933f59-ccf7-4fbe-a321-8dc19edb3a32 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:26.535984 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:26.535984 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:26.535984 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:26.535984 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:26.535984 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:26.535984 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:26.535984 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:26.535984 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:26.535984 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:26.535984 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:26.535984 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:26.535984 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:26.535984 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:26.535984 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:26.537397 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:26.537397 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:26.537397 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:26.537397 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:26.537397 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:26.537397 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:26.537397 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:26.537397 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:26.537397 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:26.537397 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:26.537397 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:26.537397 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:26.537397 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:26.537397 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:26.537397 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:26.537397 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:26.537397 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:26.538368 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:26.538368 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:26.538368 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:26.538368 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:26.545891 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 394/782] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:26.548009 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24933f59-ccf7-4fbe-a321-8dc19edb3a32 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:26.570247 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:26.656050 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90fe1e7-0351-42d9-9a0c-de1d969e9209 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:26.695989 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c733af2b-6b9e-48a6-89af-d2a7b89c9fbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:26.696571 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c733af2b-6b9e-48a6-89af-d2a7b89c9fbb None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:26.696855 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c733af2b-6b9e-48a6-89af-d2a7b89c9fbb None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:26.697133 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c733af2b-6b9e-48a6-89af-d2a7b89c9fbb None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:26.697839 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c733af2b-6b9e-48a6-89af-d2a7b89c9fbb None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:26.698146 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c733af2b-6b9e-48a6-89af-d2a7b89c9fbb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:26.702684 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c733af2b-6b9e-48a6-89af-d2a7b89c9fbb None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:26.716321 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 389/783] 10.222.0.16 () {56 vars in 1124 bytes} [Tue Jul 12 17:20:26 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:26.722033 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c733af2b-6b9e-48a6-89af-d2a7b89c9fbb None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:26.772390 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24933f59-ccf7-4fbe-a321-8dc19edb3a32 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:26.814782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:26.858874 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:26.863128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c733af2b-6b9e-48a6-89af-d2a7b89c9fbb None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:26.863258 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a972c67c-e4a7-4c47-b584-4492345eddeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:26.863817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a972c67c-e4a7-4c47-b584-4492345eddeb None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:26.864110 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a972c67c-e4a7-4c47-b584-4492345eddeb None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:26.864398 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a972c67c-e4a7-4c47-b584-4492345eddeb None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:26.865139 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a972c67c-e4a7-4c47-b584-4492345eddeb None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:26.865458 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a972c67c-e4a7-4c47-b584-4492345eddeb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:26.866886 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a972c67c-e4a7-4c47-b584-4492345eddeb None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:26.897008 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 395/784] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 350 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:26.903284 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a972c67c-e4a7-4c47-b584-4492345eddeb None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:26.905726 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02275d1c-9698-4f01-84b0-d3f7304a9a76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:26.906398 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02275d1c-9698-4f01-84b0-d3f7304a9a76 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:26.906768 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02275d1c-9698-4f01-84b0-d3f7304a9a76 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:26.907046 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02275d1c-9698-4f01-84b0-d3f7304a9a76 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:26.907928 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02275d1c-9698-4f01-84b0-d3f7304a9a76 None admin] RBAC: Adding query filter params (interface=public, service_id=28344f6ff5de47968db2c3a23a92423e, region_id=RegionOne) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:26.908248 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02275d1c-9698-4f01-84b0-d3f7304a9a76 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:26.909635 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02275d1c-9698-4f01-84b0-d3f7304a9a76 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:26.915919 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 390/785] 10.222.0.16 () {56 vars in 1251 bytes} [Tue Jul 12 17:20:26 2022] GET /identity/v3/endpoints?service_id=28344f6ff5de47968db2c3a23a92423e&interface=public®ion_id=RegionOne => generated 194 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:20:26.921988 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02275d1c-9698-4f01-84b0-d3f7304a9a76 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:27.043043 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a972c67c-e4a7-4c47-b584-4492345eddeb None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:27.046340 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02275d1c-9698-4f01-84b0-d3f7304a9a76 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:27.080624 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f413cc1-4115-4c08-9221-1aca7fe28398 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:27.081427 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f413cc1-4115-4c08-9221-1aca7fe28398 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:27.081714 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f413cc1-4115-4c08-9221-1aca7fe28398 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:27.081997 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f413cc1-4115-4c08-9221-1aca7fe28398 None admin] PATH_INFO: `/v3/users/swift` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:27.085763 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16fccf34-afed-4e05-9ee8-11cb3c1087c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:27.086442 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fccf34-afed-4e05-9ee8-11cb3c1087c9 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:27.086735 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fccf34-afed-4e05-9ee8-11cb3c1087c9 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:27.087012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fccf34-afed-4e05-9ee8-11cb3c1087c9 None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:27.087699 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16fccf34-afed-4e05-9ee8-11cb3c1087c9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:27.089059 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16fccf34-afed-4e05-9ee8-11cb3c1087c9 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:27.089712 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f413cc1-4115-4c08-9221-1aca7fe28398 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:27.091181 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f413cc1-4115-4c08-9221-1aca7fe28398 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:27.096419 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 391/786] 10.222.0.16 () {56 vars in 1087 bytes} [Tue Jul 12 17:20:26 2022] GET /identity/v3/services => generated 772 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:27.106910 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-6f413cc1-4115-4c08-9221-1aca7fe28398 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 12 17:20:27.106910 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:27.106910 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:27.106910 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:27.106910 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:27.106910 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:27.106910 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:27.106910 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:27.106910 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:27.106910 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:27.106910 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:27.106910 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:27.106910 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:20:27.106910 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:20:27.106910 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:20:27.106910 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:20:27.107952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:27.107952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:27.107952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:20:27.107952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:27.107952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:20:27.107952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:27.107952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:27.107952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:27.107952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:27.107952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:27.107952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:27.107952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:27.107952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:27.107952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:27.107952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:27.107952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:27.107952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:27.107952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:27.108995 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:27.108995 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:27.108995 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:20:27.108995 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:20:27.108995 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:20:27.108995 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:20:27.108995 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:20:27.108995 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:20:27.108995 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 12 17:20:27.108995 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:27.108995 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 396/787] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Jul 12 17:20:26 2022] GET /identity/v3/users/swift => generated 83 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:20:27.115717 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16fccf34-afed-4e05-9ee8-11cb3c1087c9 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:27.151519 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:27.239423 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16fccf34-afed-4e05-9ee8-11cb3c1087c9 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:27.280242 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f66c242-a0c1-4226-aea6-bf648bfa18fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:27.280970 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f66c242-a0c1-4226-aea6-bf648bfa18fe None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:27.281225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f66c242-a0c1-4226-aea6-bf648bfa18fe None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:27.281504 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f66c242-a0c1-4226-aea6-bf648bfa18fe None admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:27.282357 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f66c242-a0c1-4226-aea6-bf648bfa18fe None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:27.282702 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f66c242-a0c1-4226-aea6-bf648bfa18fe None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:27.284146 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f66c242-a0c1-4226-aea6-bf648bfa18fe None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:27.309103 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 392/788] 10.222.0.16 () {56 vars in 1135 bytes} [Tue Jul 12 17:20:27 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:27.314924 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f413cc1-4115-4c08-9221-1aca7fe28398 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:27.356580 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:27.432176 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f413cc1-4115-4c08-9221-1aca7fe28398 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:27.470690 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ad3577-f9e3-4ed2-9a0c-caef65307b0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:27.471494 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ad3577-f9e3-4ed2-9a0c-caef65307b0b None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:27.471779 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ad3577-f9e3-4ed2-9a0c-caef65307b0b None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:27.472055 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ad3577-f9e3-4ed2-9a0c-caef65307b0b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:27.484645 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ad3577-f9e3-4ed2-9a0c-caef65307b0b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:27.486099 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ad3577-f9e3-4ed2-9a0c-caef65307b0b None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:27.505563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-c9ad3577-f9e3-4ed2-9a0c-caef65307b0b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:27.505563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:27.505563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:27.505563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:27.505563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:27.505563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:27.505563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:27.505563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:27.505563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:27.505563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:27.505563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:27.505563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:27.505563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:27.505563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:27.505563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:27.507788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:27.507788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:27.507788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:27.507788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:27.507788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:27.507788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:27.507788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:27.507788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:27.507788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:27.507788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:27.507788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:27.507788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:27.507788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:27.507788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:27.507788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:27.507788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:27.507788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:27.509922 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:27.509922 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:27.509922 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:27.509922 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:27.514475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 397/789] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:27.515741 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f66c242-a0c1-4226-aea6-bf648bfa18fe None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:27.659326 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:27.678990 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f66c242-a0c1-4226-aea6-bf648bfa18fe None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:27.718616 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6933f20-3f3a-4dfb-811f-52d9d94d854c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:27.719690 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6933f20-3f3a-4dfb-811f-52d9d94d854c None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:27.720187 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6933f20-3f3a-4dfb-811f-52d9d94d854c None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:27.720673 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6933f20-3f3a-4dfb-811f-52d9d94d854c None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:27.721624 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6933f20-3f3a-4dfb-811f-52d9d94d854c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:27.722137 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6933f20-3f3a-4dfb-811f-52d9d94d854c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:27.723737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6933f20-3f3a-4dfb-811f-52d9d94d854c None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:27.743465 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 393/790] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:27.751512 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ad3577-f9e3-4ed2-9a0c-caef65307b0b None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:27.871209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:27.890908 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ad3577-f9e3-4ed2-9a0c-caef65307b0b None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:27.938874 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c157555-8dd2-4dce-a400-20ffddc22dd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:27.939685 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c157555-8dd2-4dce-a400-20ffddc22dd5 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:27.939971 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c157555-8dd2-4dce-a400-20ffddc22dd5 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:27.940247 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c157555-8dd2-4dce-a400-20ffddc22dd5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:27.955327 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c157555-8dd2-4dce-a400-20ffddc22dd5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:27.956782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c157555-8dd2-4dce-a400-20ffddc22dd5 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:27.977926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-2c157555-8dd2-4dce-a400-20ffddc22dd5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:27.977926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:27.977926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:27.977926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:27.977926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:27.977926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:27.977926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:27.977926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:27.977926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:27.977926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:27.977926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:27.977926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:27.977926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:20:27.977926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:20:27.977926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:20:27.978926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:20:27.978926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:27.978926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:27.978926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:27.978926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:27.978926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:27.978926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:27.978926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:27.978926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:27.978926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:27.978926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:27.978926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:27.978926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:27.978926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:27.978926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:27.978926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:27.978926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:27.979797 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:20:27.979797 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:20:27.979797 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:20:27.979797 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:20:27.979797 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:20:27.979797 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:20:27.979797 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:27.979797 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:27.983783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 398/791] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:20:27 2022] GET /identity/v3/projects/service => generated 88 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:27.991550 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6933f20-3f3a-4dfb-811f-52d9d94d854c None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:28.152269 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6933f20-3f3a-4dfb-811f-52d9d94d854c None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:28.197228 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce0db13-2e0f-44bd-99ea-b4553a9b2090 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:28.198276 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce0db13-2e0f-44bd-99ea-b4553a9b2090 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:28.198783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce0db13-2e0f-44bd-99ea-b4553a9b2090 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:28.199280 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce0db13-2e0f-44bd-99ea-b4553a9b2090 None admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:28.200415 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce0db13-2e0f-44bd-99ea-b4553a9b2090 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:28.201728 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce0db13-2e0f-44bd-99ea-b4553a9b2090 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:28.203572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce0db13-2e0f-44bd-99ea-b4553a9b2090 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:28.210339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31096b7a-a927-487d-b723-d37928672f39 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:28.210681 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31096b7a-a927-487d-b723-d37928672f39 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:28.211087 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31096b7a-a927-487d-b723-d37928672f39 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:28.212773 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 399/792] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:28.218952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98632d58-8558-4bf4-815b-aa68f1608412 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:28.219092 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98632d58-8558-4bf4-815b-aa68f1608412 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:28.219381 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98632d58-8558-4bf4-815b-aa68f1608412 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:28.223370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 394/793] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Jul 12 17:20:27 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:28.230237 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce0db13-2e0f-44bd-99ea-b4553a9b2090 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:28.239042 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-98632d58-8558-4bf4-815b-aa68f1608412 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:28.240242 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-98632d58-8558-4bf4-815b-aa68f1608412 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:28.243120 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:28.368099 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 400/794] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:28 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 12 17:20:28.370220 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7043447f-32cd-4c77-912c-9378284b4956 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:28.370802 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7043447f-32cd-4c77-912c-9378284b4956 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:28.371089 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7043447f-32cd-4c77-912c-9378284b4956 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:28.373811 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 401/795] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:28 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:28.379467 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a486c150-314a-453f-9996-a7bf90d2cde4 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:28.379760 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a486c150-314a-453f-9996-a7bf90d2cde4 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:28.380044 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a486c150-314a-453f-9996-a7bf90d2cde4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:28.383771 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce0db13-2e0f-44bd-99ea-b4553a9b2090 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:28.408025 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-a486c150-314a-453f-9996-a7bf90d2cde4 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:28.410340 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-a486c150-314a-453f-9996-a7bf90d2cde4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:28.436803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a00741e-af31-4e63-a4dc-e9282bc04805 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:28.437725 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a00741e-af31-4e63-a4dc-e9282bc04805 None admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:28.438009 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a00741e-af31-4e63-a4dc-e9282bc04805 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:28.438289 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a00741e-af31-4e63-a4dc-e9282bc04805 None admin] PATH_INFO: `/v3/projects/fd4ed2fe037c4fb7ba49c724eb9414c2/users/d7906d1a19c642bd93507a27bf3e3dae/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:28.440302 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a00741e-af31-4e63-a4dc-e9282bc04805 None admin] RBAC: Authorizing `identity:create_grant(project_id=fd4ed2fe037c4fb7ba49c724eb9414c2, user_id=d7906d1a19c642bd93507a27bf3e3dae, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:28.441748 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a00741e-af31-4e63-a4dc-e9282bc04805 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:28.490709 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 402/796] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:28 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:20:28.491551 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a486c150-314a-453f-9996-a7bf90d2cde4 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:28.506838 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:28.521462 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 395/797] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Jul 12 17:20:28 2022] PUT /identity/v3/projects/fd4ed2fe037c4fb7ba49c724eb9414c2/users/d7906d1a19c642bd93507a27bf3e3dae/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 292 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:20:28.523167 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511aac0e-bf8c-4625-a022-4902e1bd6e2b None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:28.523459 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511aac0e-bf8c-4625-a022-4902e1bd6e2b None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:28.523744 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511aac0e-bf8c-4625-a022-4902e1bd6e2b None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:28.525231 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 396/798] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:28.525887 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511aac0e-bf8c-4625-a022-4902e1bd6e2b None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:28.621310 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:28.632533 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a486c150-314a-453f-9996-a7bf90d2cde4 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:28.673647 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06870bd-f6d3-4da3-b720-3834ce70f5db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:28.674876 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06870bd-f6d3-4da3-b720-3834ce70f5db None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:28.675374 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06870bd-f6d3-4da3-b720-3834ce70f5db None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:28.676004 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06870bd-f6d3-4da3-b720-3834ce70f5db None admin] PATH_INFO: `/v3/roles/service` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:28.683935 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511aac0e-bf8c-4625-a022-4902e1bd6e2b None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:28.694980 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06870bd-f6d3-4da3-b720-3834ce70f5db None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:20:28.703468 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-f06870bd-f6d3-4da3-b720-3834ce70f5db None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:20:28.703468 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:28.703468 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:28.703468 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:28.703468 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:28.703468 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:28.703468 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:28.703468 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:28.703468 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:28.703468 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:28.703468 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:28.703468 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:28.703468 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:20:28.703468 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:20:28.703468 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:20:28.703468 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:20:28.704456 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:20:28.704456 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:20:28.704456 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:20:28.704456 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:20:28.704456 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:28.704456 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:28.704456 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:28.704456 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:28.704456 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:28.704456 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:28.704456 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:28.704456 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:28.704456 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:28.704456 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:28.704456 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:28.704456 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:28.705361 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:28.705361 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:28.705361 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:28.705361 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:28.705361 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:20:28.705361 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:20:28.705361 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:20:28.705361 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:20:28.705361 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:20:28.705361 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:20:28.705361 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:20:28.705361 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:28.705361 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 403/799] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Jul 12 17:20:28 2022] GET /identity/v3/roles/service => generated 85 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:28.706087 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9431324f-e148-4916-a753-d294882d4ea3 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:28.706155 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9431324f-e148-4916-a753-d294882d4ea3 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:28.706778 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9431324f-e148-4916-a753-d294882d4ea3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:28.736467 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0f39c6-c70c-4397-b1b3-8ee49d78555b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:28.737200 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-9431324f-e148-4916-a753-d294882d4ea3 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:28.737322 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0f39c6-c70c-4397-b1b3-8ee49d78555b None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:28.737548 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0f39c6-c70c-4397-b1b3-8ee49d78555b None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:28.737832 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0f39c6-c70c-4397-b1b3-8ee49d78555b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:28.739260 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-9431324f-e148-4916-a753-d294882d4ea3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:28.751461 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b0f39c6-c70c-4397-b1b3-8ee49d78555b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:20:28.761581 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-7b0f39c6-c70c-4397-b1b3-8ee49d78555b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:20:28.761581 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:28.761581 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:28.761581 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:28.761581 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:28.761581 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:28.761581 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:28.761581 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:28.761581 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:28.761581 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:28.761581 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:28.761581 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:28.761581 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:20:28.761581 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:20:28.761581 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:20:28.761581 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:20:28.762576 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:20:28.762576 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:20:28.762576 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:20:28.762576 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:20:28.762576 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:28.762576 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:28.762576 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:28.762576 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:28.762576 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:28.762576 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:28.762576 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:28.762576 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:28.762576 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:28.762576 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:28.762576 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:28.762576 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:28.764069 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:28.764069 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:28.764069 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:28.764069 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:28.764069 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:20:28.764069 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:20:28.764069 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:20:28.764069 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:20:28.764069 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:20:28.764069 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:20:28.764069 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:20:28.764069 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:28.766514 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 397/800] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Jul 12 17:20:28 2022] GET /identity/v3/roles/admin => generated 83 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:28.767622 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0f39c6-c70c-4397-b1b3-8ee49d78555b None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:28.828001 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:28.833754 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 404/801] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:28 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:20:28.834817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9431324f-e148-4916-a753-d294882d4ea3 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:28.898808 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0f39c6-c70c-4397-b1b3-8ee49d78555b None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:28.926146 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:28.938561 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16f9208-f02d-4f27-a707-b25660a34676 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:28.942601 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16f9208-f02d-4f27-a707-b25660a34676 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:28.942601 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16f9208-f02d-4f27-a707-b25660a34676 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:28.942601 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f16f9208-f02d-4f27-a707-b25660a34676 None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:28.942601 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f16f9208-f02d-4f27-a707-b25660a34676 None admin] RBAC: Adding query filter params (name=service) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:28.942601 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f16f9208-f02d-4f27-a707-b25660a34676 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:28.943381 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f16f9208-f02d-4f27-a707-b25660a34676 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:28.947830 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9431324f-e148-4916-a753-d294882d4ea3 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:28.953490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 398/802] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Jul 12 17:20:28 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:28.954601 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16f9208-f02d-4f27-a707-b25660a34676 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:28.992940 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d27ceae-f555-4dfb-be47-fb65302bdc37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:28.993826 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d27ceae-f555-4dfb-be47-fb65302bdc37 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:28.994158 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d27ceae-f555-4dfb-be47-fb65302bdc37 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:28.994398 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d27ceae-f555-4dfb-be47-fb65302bdc37 None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:28.995184 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d27ceae-f555-4dfb-be47-fb65302bdc37 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:28.995497 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d27ceae-f555-4dfb-be47-fb65302bdc37 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:28.997420 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d27ceae-f555-4dfb-be47-fb65302bdc37 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:29.011880 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 405/803] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:20:28 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:29.013537 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d27ceae-f555-4dfb-be47-fb65302bdc37 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:29.074000 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f16f9208-f02d-4f27-a707-b25660a34676 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:29.096150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:29.114246 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa691df0-39da-476e-b115-e7bf69312c7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:29.115436 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa691df0-39da-476e-b115-e7bf69312c7c None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:29.115929 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa691df0-39da-476e-b115-e7bf69312c7c None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:29.116408 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa691df0-39da-476e-b115-e7bf69312c7c None admin] PATH_INFO: `/v3/services/network` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:29.117263 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa691df0-39da-476e-b115-e7bf69312c7c None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:29.118879 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa691df0-39da-476e-b115-e7bf69312c7c None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:29.132528 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-fa691df0-39da-476e-b115-e7bf69312c7c None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Jul 12 17:20:29.132528 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:29.132528 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 12 17:20:29.132528 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 12 17:20:29.132528 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 12 17:20:29.132528 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 12 17:20:29.132528 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 12 17:20:29.132528 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:29.132528 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 12 17:20:29.132528 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:29.132528 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 12 17:20:29.132528 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:29.132528 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:29.132528 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:29.132528 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:29.132528 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:29.132528 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:29.133523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:29.133523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:29.133523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:29.133523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:29.133523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:29.133523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:29.133523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 12 17:20:29.133523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 12 17:20:29.133523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 12 17:20:29.133523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 12 17:20:29.133523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:29.133523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:29.133523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:29.133523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:29.133523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:29.133523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:29.134349 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:29.134349 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:29.134349 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:29.134349 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:29.134349 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:29.134349 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:29.134349 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:29.134349 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:29.134349 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:29.134349 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:29.134349 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 12 17:20:29.134349 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:29.134349 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 12 17:20:29.134349 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:29.137008 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 399/804] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:20:28 2022] GET /identity/v3/services/network => generated 88 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:29.138145 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa691df0-39da-476e-b115-e7bf69312c7c None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:29.152302 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d27ceae-f555-4dfb-be47-fb65302bdc37 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:29.193711 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:29.200178 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703d2b55-eac7-421f-b73c-4ec47e1618ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:29.201432 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703d2b55-eac7-421f-b73c-4ec47e1618ba None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:29.201723 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703d2b55-eac7-421f-b73c-4ec47e1618ba None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:29.202040 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703d2b55-eac7-421f-b73c-4ec47e1618ba None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:29.216728 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703d2b55-eac7-421f-b73c-4ec47e1618ba None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:29.219214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703d2b55-eac7-421f-b73c-4ec47e1618ba None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:29.240158 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-703d2b55-eac7-421f-b73c-4ec47e1618ba None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:29.240158 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:29.240158 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:29.240158 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:29.240158 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:29.240158 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:29.240158 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:29.240158 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:29.240158 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:29.240158 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:29.240158 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:29.240158 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:29.240158 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:29.240158 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:29.240158 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:29.241318 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:29.241318 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:29.241318 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:29.241318 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:29.241318 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:29.241318 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:29.241318 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:29.241318 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:29.241318 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:29.241318 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:29.241318 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:29.241318 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:29.241318 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:29.241318 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:29.241318 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:29.241318 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:29.241318 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:29.242412 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:29.242412 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:29.242412 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:29.242412 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:29.246195 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 406/805] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:29.247952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703d2b55-eac7-421f-b73c-4ec47e1618ba None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:29.253751 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa691df0-39da-476e-b115-e7bf69312c7c None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:29.289887 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8c3486-b445-4816-a5b6-d55afc1c04d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:29.290993 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8c3486-b445-4816-a5b6-d55afc1c04d1 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:29.291435 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8c3486-b445-4816-a5b6-d55afc1c04d1 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:29.291863 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8c3486-b445-4816-a5b6-d55afc1c04d1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:29.304238 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b8c3486-b445-4816-a5b6-d55afc1c04d1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:29.305946 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b8c3486-b445-4816-a5b6-d55afc1c04d1 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:29.324111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-0b8c3486-b445-4816-a5b6-d55afc1c04d1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:29.324111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:29.324111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:29.324111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:29.324111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:29.324111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:29.324111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:29.324111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:29.324111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:29.324111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:29.324111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:29.324111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:29.324111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:29.324111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:29.324111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:29.325235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:29.325235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:29.325235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:29.325235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:29.325235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:29.325235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:29.325235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:29.325235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:29.325235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:29.325235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:29.325235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:29.325235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:29.325235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:29.325235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:29.325235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:29.325235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:29.325235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:29.326272 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:29.326272 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:29.326272 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:29.326272 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:29.332439 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 400/806] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:29.333582 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8c3486-b445-4816-a5b6-d55afc1c04d1 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:29.366950 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703d2b55-eac7-421f-b73c-4ec47e1618ba None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:29.381573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:29.406264 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68948d5f-5468-4c62-8274-63f1f4116d28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:29.406972 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68948d5f-5468-4c62-8274-63f1f4116d28 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:29.407263 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68948d5f-5468-4c62-8274-63f1f4116d28 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:29.407550 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68948d5f-5468-4c62-8274-63f1f4116d28 None admin] PATH_INFO: `/v3/services` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:29.408448 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68948d5f-5468-4c62-8274-63f1f4116d28 None admin] RBAC: Adding query filter params (name=network) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:29.408770 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68948d5f-5468-4c62-8274-63f1f4116d28 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:29.410268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68948d5f-5468-4c62-8274-63f1f4116d28 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:29.415715 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 407/807] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Jul 12 17:20:29 2022] GET /identity/v3/services?name=network => generated 124 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:29.416579 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68948d5f-5468-4c62-8274-63f1f4116d28 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:29.447719 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8c3486-b445-4816-a5b6-d55afc1c04d1 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:29.449718 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:29.485372 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc0a77e-8a10-4f31-b6e9-83600420e7b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:29.486490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc0a77e-8a10-4f31-b6e9-83600420e7b6 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:29.486983 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc0a77e-8a10-4f31-b6e9-83600420e7b6 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:29.487461 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc0a77e-8a10-4f31-b6e9-83600420e7b6 None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:29.488392 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc0a77e-8a10-4f31-b6e9-83600420e7b6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:29.488906 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc0a77e-8a10-4f31-b6e9-83600420e7b6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:29.490516 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc0a77e-8a10-4f31-b6e9-83600420e7b6 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:29.503982 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 401/808] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:29.505152 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc0a77e-8a10-4f31-b6e9-83600420e7b6 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:29.545402 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68948d5f-5468-4c62-8274-63f1f4116d28 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:29.586265 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51fd04c0-b87e-45bc-94dd-199b89bd3082 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:29.587026 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fd04c0-b87e-45bc-94dd-199b89bd3082 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:29.587318 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fd04c0-b87e-45bc-94dd-199b89bd3082 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:29.587725 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fd04c0-b87e-45bc-94dd-199b89bd3082 None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:29.588567 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51fd04c0-b87e-45bc-94dd-199b89bd3082 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:29.588798 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51fd04c0-b87e-45bc-94dd-199b89bd3082 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:29.590184 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51fd04c0-b87e-45bc-94dd-199b89bd3082 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:29.609496 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 408/809] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:29.609496 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51fd04c0-b87e-45bc-94dd-199b89bd3082 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:29.622245 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc0a77e-8a10-4f31-b6e9-83600420e7b6 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:29.645160 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:29.660742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2664353-f5b6-458d-9a78-a8e0c5ace6b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:29.661588 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2664353-f5b6-458d-9a78-a8e0c5ace6b5 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:29.662148 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2664353-f5b6-458d-9a78-a8e0c5ace6b5 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:29.662565 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2664353-f5b6-458d-9a78-a8e0c5ace6b5 None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:29.663397 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2664353-f5b6-458d-9a78-a8e0c5ace6b5 None admin] RBAC: Adding query filter params (type=network) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:29.663810 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2664353-f5b6-458d-9a78-a8e0c5ace6b5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:29.665332 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2664353-f5b6-458d-9a78-a8e0c5ace6b5 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:29.673041 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 402/810] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Jul 12 17:20:29 2022] GET /identity/v3/services?type=network => generated 350 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:29.674230 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2664353-f5b6-458d-9a78-a8e0c5ace6b5 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:29.708801 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:29.792839 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2664353-f5b6-458d-9a78-a8e0c5ace6b5 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:29.830363 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ca8b90-762f-4015-855d-be2a48ea8cc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:29.831341 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ca8b90-762f-4015-855d-be2a48ea8cc4 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:29.831639 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ca8b90-762f-4015-855d-be2a48ea8cc4 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:29.831921 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ca8b90-762f-4015-855d-be2a48ea8cc4 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:29.839316 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ca8b90-762f-4015-855d-be2a48ea8cc4 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:29.840751 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0ca8b90-762f-4015-855d-be2a48ea8cc4 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:29.843265 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51fd04c0-b87e-45bc-94dd-199b89bd3082 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:29.855916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-b0ca8b90-762f-4015-855d-be2a48ea8cc4 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 12 17:20:29.855916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:29.855916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:29.855916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:29.855916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:29.855916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:29.855916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:29.855916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:29.855916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:29.855916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:29.855916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:29.855916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:29.855916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:20:29.855916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:20:29.855916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:20:29.855916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:20:29.857438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:29.857438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:29.857438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:20:29.857438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:29.857438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:20:29.857438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:29.857438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:29.857438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:29.857438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:29.857438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:29.857438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:29.857438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:29.857438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:29.857438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:29.857438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:29.857438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:29.857438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:29.857438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:29.858580 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:29.858580 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:29.858580 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:20:29.858580 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:20:29.858580 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:20:29.858580 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:20:29.858580 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:20:29.858580 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:20:29.858580 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 12 17:20:29.858580 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:29.858580 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 403/811] 10.222.0.16 () {56 vars in 1093 bytes} [Tue Jul 12 17:20:29 2022] GET /identity/v3/users/nova => generated 82 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:29.859907 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ca8b90-762f-4015-855d-be2a48ea8cc4 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:29.885129 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4f1754-6c91-4670-ae99-559dcb085199 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:29.885935 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4f1754-6c91-4670-ae99-559dcb085199 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:29.886219 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4f1754-6c91-4670-ae99-559dcb085199 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:29.886525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4f1754-6c91-4670-ae99-559dcb085199 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:29.894590 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4f1754-6c91-4670-ae99-559dcb085199 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:29.896175 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c4f1754-6c91-4670-ae99-559dcb085199 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:29.911166 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-2c4f1754-6c91-4670-ae99-559dcb085199 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 12 17:20:29.911166 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:29.911166 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:29.911166 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:29.911166 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:29.911166 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:29.911166 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:29.911166 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:29.911166 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:29.911166 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:29.911166 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:29.911166 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:29.911166 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:20:29.911166 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:20:29.911166 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:20:29.911166 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:20:29.912384 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:29.912384 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:29.912384 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:20:29.912384 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:29.912384 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:20:29.912384 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:29.912384 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:29.912384 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:29.912384 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:29.912384 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:29.912384 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:29.912384 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:29.912384 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:29.912384 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:29.912384 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:29.912384 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:29.912384 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:29.912384 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:29.913325 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:29.913325 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:29.913325 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:20:29.913325 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:20:29.913325 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:20:29.913325 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:20:29.913325 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:20:29.913325 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:20:29.913325 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 12 17:20:29.913325 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:29.913325 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 409/812] 10.222.0.16 () {56 vars in 1117 bytes} [Tue Jul 12 17:20:29 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 307 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:29.913920 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a630a4a-5ebc-4228-b10f-c51b3d6ce65a None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:29.913920 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a630a4a-5ebc-4228-b10f-c51b3d6ce65a None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:29.914134 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a630a4a-5ebc-4228-b10f-c51b3d6ce65a None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:29.915533 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 410/813] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:29 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:29.920957 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a630a4a-5ebc-4228-b10f-c51b3d6ce65a None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:29.960673 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:29.978064 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ca8b90-762f-4015-855d-be2a48ea8cc4 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:30.016158 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1542b811-e487-4af3-a495-e4510a4e2547 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:30.016901 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1542b811-e487-4af3-a495-e4510a4e2547 None admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:30.017185 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1542b811-e487-4af3-a495-e4510a4e2547 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:30.017463 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1542b811-e487-4af3-a495-e4510a4e2547 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:30.018199 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1542b811-e487-4af3-a495-e4510a4e2547 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:30.019597 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1542b811-e487-4af3-a495-e4510a4e2547 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:30.024061 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:30.031643 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a630a4a-5ebc-4228-b10f-c51b3d6ce65a None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:30.065831 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5530fe78-17ce-4752-b79c-3bb2f06cc244 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:30.066545 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5530fe78-17ce-4752-b79c-3bb2f06cc244 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:30.066832 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5530fe78-17ce-4752-b79c-3bb2f06cc244 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:30.067112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5530fe78-17ce-4752-b79c-3bb2f06cc244 None admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:30.067942 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5530fe78-17ce-4752-b79c-3bb2f06cc244 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:30.068250 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5530fe78-17ce-4752-b79c-3bb2f06cc244 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:30.069728 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5530fe78-17ce-4752-b79c-3bb2f06cc244 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:30.088478 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 411/814] 10.222.0.16 () {56 vars in 1133 bytes} [Tue Jul 12 17:20:29 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:30.089295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5530fe78-17ce-4752-b79c-3bb2f06cc244 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:30.183030 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 404/815] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Jul 12 17:20:29 2022] POST /identity/v3/endpoints => generated 327 bytes in 324 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:20:30.184846 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1a91b8-8ed7-4e51-8c70-625a973fd33c None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:30.185662 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1a91b8-8ed7-4e51-8c70-625a973fd33c None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:30.186036 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1a91b8-8ed7-4e51-8c70-625a973fd33c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:30.199481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5530fe78-17ce-4752-b79c-3bb2f06cc244 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:30.214132 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-4a1a91b8-8ed7-4e51-8c70-625a973fd33c None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:30.216444 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-4a1a91b8-8ed7-4e51-8c70-625a973fd33c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:30.252562 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22cc26b-67c0-4bab-97eb-94e36e470efe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:30.253258 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22cc26b-67c0-4bab-97eb-94e36e470efe None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:30.253549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22cc26b-67c0-4bab-97eb-94e36e470efe None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:30.253826 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22cc26b-67c0-4bab-97eb-94e36e470efe None admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:30.254910 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22cc26b-67c0-4bab-97eb-94e36e470efe None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:30.255223 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22cc26b-67c0-4bab-97eb-94e36e470efe None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:30.256938 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22cc26b-67c0-4bab-97eb-94e36e470efe None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:30.288364 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 412/816] 10.222.0.16 () {56 vars in 1149 bytes} [Tue Jul 12 17:20:30 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:30.289579 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22cc26b-67c0-4bab-97eb-94e36e470efe None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:30.308352 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:30.322604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 405/817] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:30 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:20:30.323502 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1a91b8-8ed7-4e51-8c70-625a973fd33c None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:30.427740 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:30.429418 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22cc26b-67c0-4bab-97eb-94e36e470efe None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:30.472180 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f392063-79ef-4594-b036-575bfe169419 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:30.472985 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1a91b8-8ed7-4e51-8c70-625a973fd33c None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:30.473356 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f392063-79ef-4594-b036-575bfe169419 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:30.473820 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f392063-79ef-4594-b036-575bfe169419 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:30.474263 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f392063-79ef-4594-b036-575bfe169419 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:30.513376 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb20b361-aaea-4156-859c-17a1fc512a93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:30.514238 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb20b361-aaea-4156-859c-17a1fc512a93 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:30.514526 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb20b361-aaea-4156-859c-17a1fc512a93 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:30.514759 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb20b361-aaea-4156-859c-17a1fc512a93 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:30.525187 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb20b361-aaea-4156-859c-17a1fc512a93 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:30.526694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb20b361-aaea-4156-859c-17a1fc512a93 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:30.545281 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-cb20b361-aaea-4156-859c-17a1fc512a93 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:30.545281 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:30.545281 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:30.545281 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:30.545281 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:30.545281 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:30.545281 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:30.545281 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:30.545281 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:30.545281 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:30.545281 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:30.545281 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:30.545281 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:30.545281 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:30.545281 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:30.546239 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:30.546239 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:30.546239 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:30.546239 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:30.546239 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:30.546239 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:30.546239 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:30.546239 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:30.546239 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:30.546239 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:30.546239 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:30.546239 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:30.546239 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:30.546239 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:30.546239 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:30.546239 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:30.546239 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:30.547209 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:30.547209 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:30.547209 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:30.547209 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:30.547209 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 406/818] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:30.547498 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb20b361-aaea-4156-859c-17a1fc512a93 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:30.555448 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f392063-79ef-4594-b036-575bfe169419 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:30.557237 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f392063-79ef-4594-b036-575bfe169419 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:30.568599 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:30.662141 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb20b361-aaea-4156-859c-17a1fc512a93 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:30.669286 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 413/819] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:20:30 2022] GET /identity/v3/auth/tokens => generated 1151 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 12 17:20:30.670381 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f392063-79ef-4594-b036-575bfe169419 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:30.687702 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:30.698960 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a380956c-dff9-40fa-b569-df5a793c84eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:30.699770 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a380956c-dff9-40fa-b569-df5a793c84eb None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:30.700060 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a380956c-dff9-40fa-b569-df5a793c84eb None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:30.700337 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a380956c-dff9-40fa-b569-df5a793c84eb None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:30.701968 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a380956c-dff9-40fa-b569-df5a793c84eb None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:30.703394 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a380956c-dff9-40fa-b569-df5a793c84eb None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:30.715346 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-a380956c-dff9-40fa-b569-df5a793c84eb None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 12 17:20:30.715346 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:30.715346 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 12 17:20:30.715346 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 12 17:20:30.715346 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 12 17:20:30.715346 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 12 17:20:30.715346 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 12 17:20:30.715346 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:30.715346 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 12 17:20:30.715346 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:30.715346 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 12 17:20:30.715346 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:30.715346 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:30.715346 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:30.715346 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:30.715346 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:30.716916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:30.716916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:30.716916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:30.716916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:30.716916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:30.716916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:30.716916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:30.716916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 12 17:20:30.716916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 12 17:20:30.716916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 12 17:20:30.716916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 12 17:20:30.716916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:30.716916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:30.716916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:30.716916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:30.716916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:30.718264 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:30.718264 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:30.718264 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:30.718264 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:30.718264 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:30.718264 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:30.718264 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:30.718264 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:30.718264 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:30.718264 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:30.718264 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:30.718264 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 12 17:20:30.718264 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:30.718264 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 12 17:20:30.718264 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:30.718264 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 407/820] 10.222.0.16 () {56 vars in 1129 bytes} [Tue Jul 12 17:20:30 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:30.719552 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83472c9f-6d75-4c6b-b0b1-d3f1e780f267 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:30.719552 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83472c9f-6d75-4c6b-b0b1-d3f1e780f267 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:30.719552 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83472c9f-6d75-4c6b-b0b1-d3f1e780f267 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:30.720672 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 408/821] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:30 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:30.721352 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83472c9f-6d75-4c6b-b0b1-d3f1e780f267 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:30.775625 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f392063-79ef-4594-b036-575bfe169419 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:30.785390 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:30.809505 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4586a7a-0a8d-4434-97d6-55544ecc61d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:30.810219 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4586a7a-0a8d-4434-97d6-55544ecc61d3 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:30.810566 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4586a7a-0a8d-4434-97d6-55544ecc61d3 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:30.810872 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4586a7a-0a8d-4434-97d6-55544ecc61d3 None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:30.811587 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4586a7a-0a8d-4434-97d6-55544ecc61d3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:30.811896 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4586a7a-0a8d-4434-97d6-55544ecc61d3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:30.813249 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4586a7a-0a8d-4434-97d6-55544ecc61d3 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:30.825699 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 414/822] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:30.826577 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83472c9f-6d75-4c6b-b0b1-d3f1e780f267 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:30.826714 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4586a7a-0a8d-4434-97d6-55544ecc61d3 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:30.865367 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b6954b-a1d1-4d6d-b7ca-d26622a723bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:30.866167 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b6954b-a1d1-4d6d-b7ca-d26622a723bd None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:30.866471 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b6954b-a1d1-4d6d-b7ca-d26622a723bd None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:30.866786 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b6954b-a1d1-4d6d-b7ca-d26622a723bd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:30.877196 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1b6954b-a1d1-4d6d-b7ca-d26622a723bd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:30.878657 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1b6954b-a1d1-4d6d-b7ca-d26622a723bd None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:30.895553 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-a1b6954b-a1d1-4d6d-b7ca-d26622a723bd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:30.895553 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:30.895553 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:30.895553 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:30.895553 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:30.895553 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:30.895553 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:30.895553 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:30.895553 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:30.895553 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:30.895553 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:30.895553 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:30.895553 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:30.895553 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:30.895553 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:30.896613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:30.896613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:30.896613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:30.896613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:30.896613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:30.896613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:30.896613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:30.896613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:30.896613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:30.896613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:30.896613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:30.896613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:30.896613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:30.896613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:30.896613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:30.896613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:30.896613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:30.897673 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:30.897673 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:30.897673 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:30.897673 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:30.897673 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 409/823] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:30.898008 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5636e9da-f0d6-4e27-88c5-7d52952eb2b8 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:30.898259 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5636e9da-f0d6-4e27-88c5-7d52952eb2b8 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:30.898568 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5636e9da-f0d6-4e27-88c5-7d52952eb2b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:30.916712 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-5636e9da-f0d6-4e27-88c5-7d52952eb2b8 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:30.917914 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-5636e9da-f0d6-4e27-88c5-7d52952eb2b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:30.932263 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4586a7a-0a8d-4434-97d6-55544ecc61d3 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:30.954408 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:30.966693 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6767d5fa-132d-4b79-a2ec-9a8da0c38205 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:30.967357 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6767d5fa-132d-4b79-a2ec-9a8da0c38205 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:30.967647 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6767d5fa-132d-4b79-a2ec-9a8da0c38205 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:30.967927 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6767d5fa-132d-4b79-a2ec-9a8da0c38205 None admin] PATH_INFO: `/v3/services` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:30.968633 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6767d5fa-132d-4b79-a2ec-9a8da0c38205 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:30.968942 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6767d5fa-132d-4b79-a2ec-9a8da0c38205 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:30.970301 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6767d5fa-132d-4b79-a2ec-9a8da0c38205 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:30.975000 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 415/824] 10.222.0.16 () {56 vars in 1124 bytes} [Tue Jul 12 17:20:30 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:30.975779 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6767d5fa-132d-4b79-a2ec-9a8da0c38205 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:30.987022 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 410/825] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:30 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:20:30.987861 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5636e9da-f0d6-4e27-88c5-7d52952eb2b8 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:31.012479 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:31.082752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6767d5fa-132d-4b79-a2ec-9a8da0c38205 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:31.088220 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5636e9da-f0d6-4e27-88c5-7d52952eb2b8 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:31.117188 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bdff980-a779-44bc-b136-344423e90343 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:31.117977 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdff980-a779-44bc-b136-344423e90343 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:31.118268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdff980-a779-44bc-b136-344423e90343 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:31.118584 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdff980-a779-44bc-b136-344423e90343 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:31.123197 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4959e7-6812-4fac-b697-5273b80fd635 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:31.123854 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4959e7-6812-4fac-b697-5273b80fd635 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:31.124139 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4959e7-6812-4fac-b697-5273b80fd635 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:31.124420 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4959e7-6812-4fac-b697-5273b80fd635 None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:31.125152 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea4959e7-6812-4fac-b697-5273b80fd635 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:31.125462 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea4959e7-6812-4fac-b697-5273b80fd635 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:31.125844 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bdff980-a779-44bc-b136-344423e90343 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:31.126881 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea4959e7-6812-4fac-b697-5273b80fd635 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:31.127287 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bdff980-a779-44bc-b136-344423e90343 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:31.139171 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 411/826] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:31.139979 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4959e7-6812-4fac-b697-5273b80fd635 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:31.140751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-2bdff980-a779-44bc-b136-344423e90343 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:31.140751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:31.140751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:31.140751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:31.140751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:31.140751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:31.140751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:31.140751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:31.140751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:31.140751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:31.140751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:31.140751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:31.140751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:20:31.140751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:20:31.140751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:20:31.141602 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:20:31.141602 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:31.141602 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:31.141602 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:31.141602 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:31.141602 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:31.141602 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:31.141602 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:31.141602 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:31.141602 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:31.141602 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:31.141602 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:31.141602 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:31.141602 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:31.141602 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:31.141602 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:31.141602 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:31.147573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:20:31.147573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:20:31.147573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:20:31.147573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:20:31.147573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:20:31.147573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:20:31.147573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:31.147573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:31.147573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 416/827] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:20:30 2022] GET /identity/v3/projects/service => generated 88 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:31.147573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bdff980-a779-44bc-b136-344423e90343 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:31.176723 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:31.237741 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:31.246109 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bdff980-a779-44bc-b136-344423e90343 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:31.246808 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4959e7-6812-4fac-b697-5273b80fd635 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:31.286025 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14036f3b-6af6-4709-9802-b70a390902aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:31.286365 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbee01af-4351-4407-a899-09ad27829206 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:31.286823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14036f3b-6af6-4709-9802-b70a390902aa None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:31.286984 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbee01af-4351-4407-a899-09ad27829206 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:31.287080 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14036f3b-6af6-4709-9802-b70a390902aa None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:31.287176 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbee01af-4351-4407-a899-09ad27829206 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:31.287267 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14036f3b-6af6-4709-9802-b70a390902aa None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:31.287404 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbee01af-4351-4407-a899-09ad27829206 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:31.287985 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14036f3b-6af6-4709-9802-b70a390902aa None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:31.288294 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14036f3b-6af6-4709-9802-b70a390902aa None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:31.289677 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14036f3b-6af6-4709-9802-b70a390902aa None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:31.295300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 412/828] 10.222.0.16 () {56 vars in 1124 bytes} [Tue Jul 12 17:20:31 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:31.296201 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14036f3b-6af6-4709-9802-b70a390902aa None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:31.300361 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbee01af-4351-4407-a899-09ad27829206 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:20:31.308425 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-fbee01af-4351-4407-a899-09ad27829206 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:20:31.308425 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:31.308425 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:31.308425 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:31.308425 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:31.308425 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:31.308425 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:31.308425 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:31.308425 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:31.308425 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:31.308425 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:31.308425 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:31.308425 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:20:31.308425 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:20:31.308425 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:20:31.308425 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:20:31.309625 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:20:31.309625 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:20:31.309625 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:20:31.309625 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:20:31.309625 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:31.309625 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:31.309625 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:31.309625 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:31.309625 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:31.309625 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:31.309625 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:31.309625 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:31.309625 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:31.309625 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:31.309625 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:31.309625 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:31.311849 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:31.311849 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:31.311849 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:31.311849 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:31.311849 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:20:31.311849 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:20:31.311849 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:20:31.311849 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:20:31.311849 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:20:31.311849 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:20:31.311849 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:20:31.311849 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:31.311849 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 417/829] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Jul 12 17:20:31 2022] GET /identity/v3/roles/admin => generated 83 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:31.311849 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbee01af-4351-4407-a899-09ad27829206 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:31.423900 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbee01af-4351-4407-a899-09ad27829206 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:31.459002 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669a7385-9195-4a55-b7c5-18ed9fb2e77a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:31.459937 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669a7385-9195-4a55-b7c5-18ed9fb2e77a None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:31.460234 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669a7385-9195-4a55-b7c5-18ed9fb2e77a None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:31.460513 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669a7385-9195-4a55-b7c5-18ed9fb2e77a None admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:31.461306 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-669a7385-9195-4a55-b7c5-18ed9fb2e77a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:31.461619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-669a7385-9195-4a55-b7c5-18ed9fb2e77a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:31.463087 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-669a7385-9195-4a55-b7c5-18ed9fb2e77a None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:31.477036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 418/830] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Jul 12 17:20:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:31.477952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669a7385-9195-4a55-b7c5-18ed9fb2e77a None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:31.494235 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:31.510159 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:31.517329 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14036f3b-6af6-4709-9802-b70a390902aa None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:31.552897 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858de3fa-adbd-410d-879b-083d71d88c05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:31.553695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858de3fa-adbd-410d-879b-083d71d88c05 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:31.553985 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858de3fa-adbd-410d-879b-083d71d88c05 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:31.554269 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-858de3fa-adbd-410d-879b-083d71d88c05 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:31.561260 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-858de3fa-adbd-410d-879b-083d71d88c05 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:31.563210 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-858de3fa-adbd-410d-879b-083d71d88c05 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:31.577857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-858de3fa-adbd-410d-879b-083d71d88c05 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:31.577857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:31.577857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:31.577857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:31.577857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:31.577857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:31.577857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:31.577857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:31.577857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:31.577857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:31.577857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:31.577857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:31.577857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:20:31.577857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:20:31.577857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:20:31.578771 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:20:31.578771 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:31.578771 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:31.578771 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:31.578771 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:31.578771 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:31.578771 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:31.578771 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:31.578771 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:31.578771 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:31.578771 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:31.578771 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:31.578771 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:31.578771 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:31.578771 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:31.578771 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:31.578771 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:31.579672 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:20:31.579672 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:20:31.579672 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:20:31.579672 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:20:31.579672 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:20:31.579672 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:20:31.579672 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:31.579672 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:31.579672 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 413/831] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:20:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 283 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:31.580182 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858de3fa-adbd-410d-879b-083d71d88c05 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:31.587072 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669a7385-9195-4a55-b7c5-18ed9fb2e77a None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:31.623646 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c93831-1647-4555-bcd0-abe284b5c5a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:31.624336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c93831-1647-4555-bcd0-abe284b5c5a9 None admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:31.624629 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c93831-1647-4555-bcd0-abe284b5c5a9 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:31.624911 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c93831-1647-4555-bcd0-abe284b5c5a9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:31.627792 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c93831-1647-4555-bcd0-abe284b5c5a9 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:31.627792 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c93831-1647-4555-bcd0-abe284b5c5a9 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:31.681310 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-858de3fa-adbd-410d-879b-083d71d88c05 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:31.703766 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 419/832] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Jul 12 17:20:31 2022] POST /identity/v3/endpoints => generated 346 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:20:31.704568 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c93831-1647-4555-bcd0-abe284b5c5a9 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:31.723119 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e4376d-e8b5-46cb-bb91-4a77c53b38ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:31.723813 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e4376d-e8b5-46cb-bb91-4a77c53b38ab None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:31.724100 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e4376d-e8b5-46cb-bb91-4a77c53b38ab None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:31.724436 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e4376d-e8b5-46cb-bb91-4a77c53b38ab None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:31.725176 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e4376d-e8b5-46cb-bb91-4a77c53b38ab None admin] RBAC: Adding query filter params (name=admin) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:31.725487 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e4376d-e8b5-46cb-bb91-4a77c53b38ab None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:31.727186 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e4376d-e8b5-46cb-bb91-4a77c53b38ab None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:31.738239 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 414/833] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:20:31 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:31.739159 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e4376d-e8b5-46cb-bb91-4a77c53b38ab None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:31.756637 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:31.811804 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:31.850193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c93831-1647-4555-bcd0-abe284b5c5a9 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:31.854755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e4376d-e8b5-46cb-bb91-4a77c53b38ab None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:31.892567 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da742e03-ed57-4e46-9931-2d3d797da463 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:31.893565 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da742e03-ed57-4e46-9931-2d3d797da463 None admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:31.893850 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da742e03-ed57-4e46-9931-2d3d797da463 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:31.894132 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da742e03-ed57-4e46-9931-2d3d797da463 None admin] PATH_INFO: `/v3/projects/fd4ed2fe037c4fb7ba49c724eb9414c2/users/3ac0d5d8de4943a3888abee3b24b8a6f/roles/bef74d4975164f75bcec96ac47afea73` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:31.895240 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6389af8a-dc00-4023-ac0f-5e61d583b607 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:31.895900 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6389af8a-dc00-4023-ac0f-5e61d583b607 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:31.896240 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6389af8a-dc00-4023-ac0f-5e61d583b607 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:31.896528 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6389af8a-dc00-4023-ac0f-5e61d583b607 None admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:31.897329 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6389af8a-dc00-4023-ac0f-5e61d583b607 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:31.897651 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6389af8a-dc00-4023-ac0f-5e61d583b607 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:31.899952 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6389af8a-dc00-4023-ac0f-5e61d583b607 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:31.912817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da742e03-ed57-4e46-9931-2d3d797da463 None admin] RBAC: Authorizing `identity:create_grant(project_id=fd4ed2fe037c4fb7ba49c724eb9414c2, user_id=3ac0d5d8de4943a3888abee3b24b8a6f, role_id=bef74d4975164f75bcec96ac47afea73)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:31.913782 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 415/834] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Jul 12 17:20:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:31.914677 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6389af8a-dc00-4023-ac0f-5e61d583b607 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:31.915173 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da742e03-ed57-4e46-9931-2d3d797da463 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:31.984711 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:32.014556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6389af8a-dc00-4023-ac0f-5e61d583b607 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:32.048557 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20197525-0353-4ce3-9d96-49cb6f7f5976 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:32.049356 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20197525-0353-4ce3-9d96-49cb6f7f5976 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:32.049657 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20197525-0353-4ce3-9d96-49cb6f7f5976 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:32.049936 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20197525-0353-4ce3-9d96-49cb6f7f5976 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:32.055329 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 420/835] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Jul 12 17:20:31 2022] PUT /identity/v3/projects/fd4ed2fe037c4fb7ba49c724eb9414c2/users/3ac0d5d8de4943a3888abee3b24b8a6f/roles/bef74d4975164f75bcec96ac47afea73 => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:20:32.056154 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da742e03-ed57-4e46-9931-2d3d797da463 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:32.070348 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20197525-0353-4ce3-9d96-49cb6f7f5976 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:32.073083 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20197525-0353-4ce3-9d96-49cb6f7f5976 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:32.102064 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-20197525-0353-4ce3-9d96-49cb6f7f5976 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:32.102064 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:32.102064 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:32.102064 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:32.102064 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:32.102064 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:32.102064 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:32.102064 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:32.102064 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:32.102064 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:32.102064 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:32.102064 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:32.102064 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:32.102064 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:32.102064 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:32.103128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:32.103128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:32.103128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:32.103128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:32.103128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:32.103128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:32.103128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:32.103128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:32.103128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:32.103128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:32.103128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:32.103128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:32.103128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:32.103128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:32.103128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:32.103128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:32.103128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:32.104029 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:32.104029 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:32.104029 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:32.104029 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:32.107405 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 416/836] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:32.113377 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20197525-0353-4ce3-9d96-49cb6f7f5976 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:32.185901 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da742e03-ed57-4e46-9931-2d3d797da463 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:32.206776 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:32.231248 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfecdd5-2483-496e-82c2-188f93bd8e3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:32.232462 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfecdd5-2483-496e-82c2-188f93bd8e3c None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:32.232744 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfecdd5-2483-496e-82c2-188f93bd8e3c None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:32.233018 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfecdd5-2483-496e-82c2-188f93bd8e3c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:32.233856 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dfecdd5-2483-496e-82c2-188f93bd8e3c None admin] RBAC: Adding query filter params (role.id=eb05bf286c894ea9bf44a6c023a4323b, scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2, user.id=9012d5b88f554dc596582a7693c2a72a) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:32.234170 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dfecdd5-2483-496e-82c2-188f93bd8e3c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:32.236325 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dfecdd5-2483-496e-82c2-188f93bd8e3c None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:32.236454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20197525-0353-4ce3-9d96-49cb6f7f5976 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:32.242419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 421/837] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:20:32 2022] GET /identity/v3/role_assignments?user.id=9012d5b88f554dc596582a7693c2a72a&scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2&role.id=eb05bf286c894ea9bf44a6c023a4323b => generated 608 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:32.265364 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:32.282226 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5acb3750-e506-4b39-b2c4-19e0abe656e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:32.282956 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5acb3750-e506-4b39-b2c4-19e0abe656e9 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:32.283242 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5acb3750-e506-4b39-b2c4-19e0abe656e9 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:32.284136 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5acb3750-e506-4b39-b2c4-19e0abe656e9 None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:32.284252 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5acb3750-e506-4b39-b2c4-19e0abe656e9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:32.284569 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5acb3750-e506-4b39-b2c4-19e0abe656e9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:32.285996 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5acb3750-e506-4b39-b2c4-19e0abe656e9 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:32.306365 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 417/838] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:32.312486 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfecdd5-2483-496e-82c2-188f93bd8e3c None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:32.426425 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfecdd5-2483-496e-82c2-188f93bd8e3c None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:32.464446 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77748119-ecaf-4fc8-8b4c-7942d202eaa9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:32.465358 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77748119-ecaf-4fc8-8b4c-7942d202eaa9 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:32.465665 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77748119-ecaf-4fc8-8b4c-7942d202eaa9 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:32.466037 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77748119-ecaf-4fc8-8b4c-7942d202eaa9 None admin] PATH_INFO: `/v3/users/swift` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:32.474895 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77748119-ecaf-4fc8-8b4c-7942d202eaa9 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:32.476447 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77748119-ecaf-4fc8-8b4c-7942d202eaa9 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:32.492719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-77748119-ecaf-4fc8-8b4c-7942d202eaa9 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 12 17:20:32.492719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:32.492719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:32.492719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:32.492719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:32.492719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:32.492719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:32.492719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:32.492719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:32.492719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:32.492719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:32.492719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:32.492719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:20:32.492719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:20:32.492719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:20:32.492719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:20:32.493722 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:32.493722 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:32.493722 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:20:32.493722 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:32.493722 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:20:32.493722 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:32.493722 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:32.493722 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:32.493722 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:32.493722 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:32.493722 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:32.493722 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:32.493722 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:32.493722 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:32.493722 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:32.493722 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:32.493722 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:32.493722 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:32.494705 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:32.494705 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:32.494705 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:20:32.494705 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:20:32.494705 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:20:32.494705 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:20:32.494705 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:20:32.494705 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:20:32.494705 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 12 17:20:32.494705 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:32.495841 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 422/839] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Jul 12 17:20:32 2022] GET /identity/v3/users/swift => generated 83 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:32.501398 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5acb3750-e506-4b39-b2c4-19e0abe656e9 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:32.611417 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5acb3750-e506-4b39-b2c4-19e0abe656e9 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:32.649753 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a437e0-8d46-4d0d-8b3b-96b0404556c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:32.650450 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a437e0-8d46-4d0d-8b3b-96b0404556c4 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:32.650746 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a437e0-8d46-4d0d-8b3b-96b0404556c4 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:32.651030 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3a437e0-8d46-4d0d-8b3b-96b0404556c4 None admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:32.652701 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a437e0-8d46-4d0d-8b3b-96b0404556c4 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:32.653024 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a437e0-8d46-4d0d-8b3b-96b0404556c4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:32.655488 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3a437e0-8d46-4d0d-8b3b-96b0404556c4 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:32.674713 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 418/840] 10.222.0.16 () {56 vars in 1135 bytes} [Tue Jul 12 17:20:32 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:32.680229 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77748119-ecaf-4fc8-8b4c-7942d202eaa9 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:32.726482 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:32.788302 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77748119-ecaf-4fc8-8b4c-7942d202eaa9 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:32.826242 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e03a91-5d71-48d7-912d-e80af1d4494f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:32.826947 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e03a91-5d71-48d7-912d-e80af1d4494f None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:32.827243 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e03a91-5d71-48d7-912d-e80af1d4494f None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:32.827528 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e03a91-5d71-48d7-912d-e80af1d4494f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:32.902858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e03a91-5d71-48d7-912d-e80af1d4494f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:32.904408 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e03a91-5d71-48d7-912d-e80af1d4494f None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:32.951561 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:33.013605 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 423/841] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:20:32 2022] GET /identity/v3/auth/tokens => generated 1151 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 12 17:20:33.019624 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a437e0-8d46-4d0d-8b3b-96b0404556c4 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:33.065879 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:33.127437 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3a437e0-8d46-4d0d-8b3b-96b0404556c4 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:33.162515 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4965ee10-337c-413f-9b62-985a69de8eb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:33.163313 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4965ee10-337c-413f-9b62-985a69de8eb7 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:33.163605 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4965ee10-337c-413f-9b62-985a69de8eb7 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:33.163887 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4965ee10-337c-413f-9b62-985a69de8eb7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:33.174285 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4965ee10-337c-413f-9b62-985a69de8eb7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:33.175757 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4965ee10-337c-413f-9b62-985a69de8eb7 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:33.193841 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-4965ee10-337c-413f-9b62-985a69de8eb7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:33.193841 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:33.193841 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:33.193841 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:33.193841 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:33.193841 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:33.193841 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:33.193841 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:33.193841 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:33.193841 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:33.193841 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:33.193841 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:33.193841 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:33.193841 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:33.193841 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:33.194803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:33.194803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:33.194803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:33.194803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:33.194803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:33.194803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:33.194803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:33.194803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:33.194803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:33.194803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:33.194803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:33.194803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:33.194803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:33.194803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:33.194803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:33.194803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:33.194803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:33.195765 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:33.195765 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:33.195765 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:33.195765 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:33.195765 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 419/842] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:33.202629 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e03a91-5d71-48d7-912d-e80af1d4494f None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:33.311415 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e03a91-5d71-48d7-912d-e80af1d4494f None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:33.351386 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb25c2d-af60-489c-94c6-6c7e56f29004 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:33.352656 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb25c2d-af60-489c-94c6-6c7e56f29004 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:33.353218 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb25c2d-af60-489c-94c6-6c7e56f29004 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:33.353750 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb25c2d-af60-489c-94c6-6c7e56f29004 None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:33.354812 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb25c2d-af60-489c-94c6-6c7e56f29004 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:33.355373 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb25c2d-af60-489c-94c6-6c7e56f29004 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:33.357517 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb25c2d-af60-489c-94c6-6c7e56f29004 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:33.374723 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 424/843] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:33.379551 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4965ee10-337c-413f-9b62-985a69de8eb7 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:33.481496 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:33.488047 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4965ee10-337c-413f-9b62-985a69de8eb7 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:33.523824 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e159b90-6fff-4884-aa43-c568db72549e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:33.524624 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e159b90-6fff-4884-aa43-c568db72549e None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:33.524915 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e159b90-6fff-4884-aa43-c568db72549e None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:33.525198 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e159b90-6fff-4884-aa43-c568db72549e None admin] PATH_INFO: `/v3/projects/service` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:33.532871 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e159b90-6fff-4884-aa43-c568db72549e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:33.534322 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e159b90-6fff-4884-aa43-c568db72549e None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:33.548808 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-6e159b90-6fff-4884-aa43-c568db72549e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:33.548808 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:33.548808 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:33.548808 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:33.548808 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:33.548808 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:33.548808 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:33.548808 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:33.548808 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:33.548808 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:33.548808 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:33.548808 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:33.548808 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:20:33.548808 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:20:33.548808 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:20:33.549715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:20:33.549715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:33.549715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:33.549715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:33.549715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:33.549715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:33.549715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:33.549715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:33.549715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:33.549715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:33.549715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:33.549715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:33.549715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:33.549715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:33.549715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:33.549715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:33.549715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:33.550877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:20:33.550877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:20:33.550877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:20:33.550877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:20:33.550877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:20:33.550877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:20:33.550877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:33.550877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:33.550877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 420/844] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:20:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:33.555915 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb25c2d-af60-489c-94c6-6c7e56f29004 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:33.563417 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:33.672399 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb25c2d-af60-489c-94c6-6c7e56f29004 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:33.709094 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b1c342-eb73-405b-9ee6-d4935b8f0514 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:33.709772 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b1c342-eb73-405b-9ee6-d4935b8f0514 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:33.710068 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b1c342-eb73-405b-9ee6-d4935b8f0514 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:33.710360 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b1c342-eb73-405b-9ee6-d4935b8f0514 None admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:33.711196 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b1c342-eb73-405b-9ee6-d4935b8f0514 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:33.711518 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b1c342-eb73-405b-9ee6-d4935b8f0514 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:33.713075 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b1c342-eb73-405b-9ee6-d4935b8f0514 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:33.726304 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 425/845] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Jul 12 17:20:33 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:33.732982 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e159b90-6fff-4884-aa43-c568db72549e None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:33.843375 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e159b90-6fff-4884-aa43-c568db72549e None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:33.879702 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-854a64f0-aa67-4ebb-a289-8061b330cac4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:33.880308 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854a64f0-aa67-4ebb-a289-8061b330cac4 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:33.880599 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854a64f0-aa67-4ebb-a289-8061b330cac4 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:33.880885 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854a64f0-aa67-4ebb-a289-8061b330cac4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:33.881730 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-854a64f0-aa67-4ebb-a289-8061b330cac4 None admin] RBAC: Adding query filter params (role.id=eb05bf286c894ea9bf44a6c023a4323b, scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2, user.id=d7906d1a19c642bd93507a27bf3e3dae) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:33.882050 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-854a64f0-aa67-4ebb-a289-8061b330cac4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:33.883644 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-854a64f0-aa67-4ebb-a289-8061b330cac4 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:33.890744 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 421/846] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:20:33 2022] GET /identity/v3/role_assignments?user.id=d7906d1a19c642bd93507a27bf3e3dae&scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2&role.id=eb05bf286c894ea9bf44a6c023a4323b => generated 608 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:33.897050 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ddb148-756e-437c-b3bb-4276b5550ed1 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:33.897354 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ddb148-756e-437c-b3bb-4276b5550ed1 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:33.897641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ddb148-756e-437c-b3bb-4276b5550ed1 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:33.900480 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 426/847] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:33.905616 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1799966a-b324-4f0c-b420-6ce30b5ad330 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:33.905932 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1799966a-b324-4f0c-b420-6ce30b5ad330 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:33.906184 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1799966a-b324-4f0c-b420-6ce30b5ad330 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:33.926153 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-1799966a-b324-4f0c-b420-6ce30b5ad330 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:33.927586 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-1799966a-b324-4f0c-b420-6ce30b5ad330 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:33.944174 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:34.013781 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 422/848] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:33 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:20:34.032252 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ddb148-756e-437c-b3bb-4276b5550ed1 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:34.105692 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:34.148259 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ddb148-756e-437c-b3bb-4276b5550ed1 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:34.184670 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb0f6cd-a9cd-4bda-bafd-98aaf8bceb83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:34.186081 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb0f6cd-a9cd-4bda-bafd-98aaf8bceb83 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:34.186396 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb0f6cd-a9cd-4bda-bafd-98aaf8bceb83 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:34.186716 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb0f6cd-a9cd-4bda-bafd-98aaf8bceb83 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:34.187382 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb0f6cd-a9cd-4bda-bafd-98aaf8bceb83 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:34.188752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb0f6cd-a9cd-4bda-bafd-98aaf8bceb83 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:34.201021 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-afb0f6cd-a9cd-4bda-bafd-98aaf8bceb83 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 12 17:20:34.201021 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:34.201021 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 12 17:20:34.201021 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 12 17:20:34.201021 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 12 17:20:34.201021 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 12 17:20:34.201021 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 12 17:20:34.201021 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:34.201021 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 12 17:20:34.201021 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:34.201021 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 12 17:20:34.201021 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:34.201021 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:34.201021 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:34.201021 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:34.201021 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:34.201021 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:34.202076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:34.202076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:34.202076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:34.202076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:34.202076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:34.202076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:34.202076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 12 17:20:34.202076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 12 17:20:34.202076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 12 17:20:34.202076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 12 17:20:34.202076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:34.202076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:34.202076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:34.202076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:34.202076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:34.202076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:34.203827 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:34.203827 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:34.203827 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:34.203827 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:34.203827 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:34.203827 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:34.203827 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:34.203827 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:34.203827 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:34.203827 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:34.203827 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 12 17:20:34.203827 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:34.203827 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 12 17:20:34.203827 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:34.206543 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 427/849] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Jul 12 17:20:34 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:34.208645 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1799966a-b324-4f0c-b420-6ce30b5ad330 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:34.285325 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0bf03c6-ad11-4c3e-b4fc-f5b370dc52af None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:34.285648 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0bf03c6-ad11-4c3e-b4fc-f5b370dc52af None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:34.285877 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0bf03c6-ad11-4c3e-b4fc-f5b370dc52af None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:34.287574 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 428/850] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:34.294386 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74c53fd-3b1d-4a98-bc2e-71b913862715 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:34.294967 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74c53fd-3b1d-4a98-bc2e-71b913862715 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:34.295452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74c53fd-3b1d-4a98-bc2e-71b913862715 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:34.313540 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-d74c53fd-3b1d-4a98-bc2e-71b913862715 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:34.314790 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-d74c53fd-3b1d-4a98-bc2e-71b913862715 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:34.319958 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1799966a-b324-4f0c-b420-6ce30b5ad330 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:34.368668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2614ac2e-4ae8-4cc4-a670-3430d3d76fae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:34.369338 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2614ac2e-4ae8-4cc4-a670-3430d3d76fae None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:34.369623 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2614ac2e-4ae8-4cc4-a670-3430d3d76fae None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:34.369902 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2614ac2e-4ae8-4cc4-a670-3430d3d76fae None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:34.370696 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2614ac2e-4ae8-4cc4-a670-3430d3d76fae None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:34.371005 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2614ac2e-4ae8-4cc4-a670-3430d3d76fae None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:34.373078 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2614ac2e-4ae8-4cc4-a670-3430d3d76fae None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:34.382683 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 423/851] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Jul 12 17:20:34 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:34.384275 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2614ac2e-4ae8-4cc4-a670-3430d3d76fae None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:34.406541 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 429/852] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:34 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 12 17:20:34.424562 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:34.424852 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d74c53fd-3b1d-4a98-bc2e-71b913862715 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:34.493522 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:34.515053 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2614ac2e-4ae8-4cc4-a670-3430d3d76fae None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:34.542474 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d74c53fd-3b1d-4a98-bc2e-71b913862715 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:34.552012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9640f1-3dae-46cf-a1c9-20dec26269c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:34.552720 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9640f1-3dae-46cf-a1c9-20dec26269c2 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:34.553011 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9640f1-3dae-46cf-a1c9-20dec26269c2 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:34.553303 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9640f1-3dae-46cf-a1c9-20dec26269c2 None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:34.554020 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b9640f1-3dae-46cf-a1c9-20dec26269c2 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:34.554334 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b9640f1-3dae-46cf-a1c9-20dec26269c2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:34.555734 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b9640f1-3dae-46cf-a1c9-20dec26269c2 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:34.570462 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 424/853] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Jul 12 17:20:34 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:34.570462 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675b0003-d5e9-4323-b7d3-5ee4b64b0277 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:34.570462 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675b0003-d5e9-4323-b7d3-5ee4b64b0277 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:34.570462 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-675b0003-d5e9-4323-b7d3-5ee4b64b0277 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:34.570462 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 425/854] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:34.570462 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ad85b9-df4c-41cc-97f6-6c4221a0edbd None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:34.570462 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ad85b9-df4c-41cc-97f6-6c4221a0edbd None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:34.571156 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10ad85b9-df4c-41cc-97f6-6c4221a0edbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:34.590327 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-10ad85b9-df4c-41cc-97f6-6c4221a0edbd None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:34.592189 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-10ad85b9-df4c-41cc-97f6-6c4221a0edbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:34.602544 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6df8c3b-558c-45c9-9cc6-f21dfa03209c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:34.604193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6df8c3b-558c-45c9-9cc6-f21dfa03209c None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:34.604741 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6df8c3b-558c-45c9-9cc6-f21dfa03209c None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:34.605642 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6df8c3b-558c-45c9-9cc6-f21dfa03209c None admin] PATH_INFO: `/v3/roles/service` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:34.623119 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6df8c3b-558c-45c9-9cc6-f21dfa03209c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:20:34.634784 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-a6df8c3b-558c-45c9-9cc6-f21dfa03209c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:20:34.634784 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:34.634784 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:34.634784 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:34.634784 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:34.634784 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:34.634784 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:34.634784 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:34.634784 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:34.634784 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:34.634784 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:34.634784 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:34.634784 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:20:34.634784 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:20:34.634784 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:20:34.634784 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:20:34.635812 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:20:34.635812 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:20:34.635812 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:20:34.635812 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:20:34.635812 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:34.635812 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:34.635812 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:34.635812 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:34.635812 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:34.635812 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:34.635812 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:34.635812 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:34.635812 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:34.635812 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:34.635812 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:34.635812 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:34.636641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:34.636641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:34.636641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:34.636641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:34.636641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:20:34.636641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:20:34.636641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:20:34.636641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:20:34.636641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:20:34.636641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:20:34.636641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:20:34.636641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:34.638539 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 430/855] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Jul 12 17:20:34 2022] GET /identity/v3/roles/service => generated 85 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:20:34.642689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6df8c3b-558c-45c9-9cc6-f21dfa03209c None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:34.678836 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 426/856] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:34 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 12 17:20:34.697693 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ad85b9-df4c-41cc-97f6-6c4221a0edbd None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:34.706830 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:34.779239 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6df8c3b-558c-45c9-9cc6-f21dfa03209c None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:34.803679 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:34.823654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3c8afc-8098-450b-a487-692aee80963d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:34.824333 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3c8afc-8098-450b-a487-692aee80963d None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:34.824623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3c8afc-8098-450b-a487-692aee80963d None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:34.824917 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3c8afc-8098-450b-a487-692aee80963d None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:34.826192 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3c8afc-8098-450b-a487-692aee80963d None admin] RBAC: Adding query filter params (name=service) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:34.826550 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3c8afc-8098-450b-a487-692aee80963d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:34.827961 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3c8afc-8098-450b-a487-692aee80963d None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:34.832911 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10ad85b9-df4c-41cc-97f6-6c4221a0edbd None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:34.837682 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 431/857] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Jul 12 17:20:34 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:20:34.848458 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3c8afc-8098-450b-a487-692aee80963d None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:34.873335 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b667e26a-95f3-431c-aea3-49884d16e1f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:34.874554 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b667e26a-95f3-431c-aea3-49884d16e1f2 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:34.875048 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b667e26a-95f3-431c-aea3-49884d16e1f2 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:34.875539 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b667e26a-95f3-431c-aea3-49884d16e1f2 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:34.876469 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b667e26a-95f3-431c-aea3-49884d16e1f2 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:34.878047 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b667e26a-95f3-431c-aea3-49884d16e1f2 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:34.895179 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-b667e26a-95f3-431c-aea3-49884d16e1f2 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 12 17:20:34.895179 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:34.895179 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 12 17:20:34.895179 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 12 17:20:34.895179 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 12 17:20:34.895179 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 12 17:20:34.895179 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 12 17:20:34.895179 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:34.895179 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 12 17:20:34.895179 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:34.895179 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 12 17:20:34.895179 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:34.895179 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:34.895179 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:34.895179 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:34.895179 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:34.896225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:34.896225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:34.896225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:34.896225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:34.896225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:34.896225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:34.896225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:34.896225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 12 17:20:34.896225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 12 17:20:34.896225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 12 17:20:34.896225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 12 17:20:34.896225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:34.896225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:34.896225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:34.896225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:34.896225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:34.897066 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:34.897066 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:34.897066 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:34.897066 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:34.897066 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:34.897066 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:34.897066 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:34.897066 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:34.897066 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:34.897066 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:34.897066 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:34.897066 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 12 17:20:34.897066 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:34.897066 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 12 17:20:34.897066 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:34.902823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 427/858] 10.222.0.16 () {56 vars in 1132 bytes} [Tue Jul 12 17:20:34 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:34.906512 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b667e26a-95f3-431c-aea3-49884d16e1f2 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:34.958180 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3c8afc-8098-450b-a487-692aee80963d None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:34.981788 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:34.997911 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f33a255-30ac-40fd-9f74-01df765111f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:34.999199 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f33a255-30ac-40fd-9f74-01df765111f1 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:35.005300 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f33a255-30ac-40fd-9f74-01df765111f1 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:35.005300 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f33a255-30ac-40fd-9f74-01df765111f1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:35.011313 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f33a255-30ac-40fd-9f74-01df765111f1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:35.013903 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f33a255-30ac-40fd-9f74-01df765111f1 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:35.015094 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b667e26a-95f3-431c-aea3-49884d16e1f2 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:35.032663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-6f33a255-30ac-40fd-9f74-01df765111f1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:35.032663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:35.032663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:35.032663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:35.032663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:35.032663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:35.032663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:35.032663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:35.032663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:35.032663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:35.032663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:35.032663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:35.032663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:35.032663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:35.032663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:35.033574 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:35.033574 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:35.033574 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:35.033574 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:35.033574 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:35.033574 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:35.033574 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:35.033574 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:35.033574 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:35.033574 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:35.033574 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:35.033574 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:35.033574 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:35.033574 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:35.033574 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:35.033574 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:35.033574 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:35.034492 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:35.034492 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:35.034492 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:35.034492 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:35.037141 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 432/859] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:35.042464 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f33a255-30ac-40fd-9f74-01df765111f1 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:35.057240 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b94d57-7366-4edb-8c2a-a0ac216c9e8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:35.058594 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b94d57-7366-4edb-8c2a-a0ac216c9e8f None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:35.059070 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b94d57-7366-4edb-8c2a-a0ac216c9e8f None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:35.060490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b94d57-7366-4edb-8c2a-a0ac216c9e8f None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:35.061708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b94d57-7366-4edb-8c2a-a0ac216c9e8f None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:35.062206 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b94d57-7366-4edb-8c2a-a0ac216c9e8f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:35.064352 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b94d57-7366-4edb-8c2a-a0ac216c9e8f None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:35.070537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 428/860] 10.222.0.16 () {56 vars in 1126 bytes} [Tue Jul 12 17:20:34 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:20:35.076448 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b94d57-7366-4edb-8c2a-a0ac216c9e8f None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:35.086182 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:35.167035 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f33a255-30ac-40fd-9f74-01df765111f1 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:35.190067 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b94d57-7366-4edb-8c2a-a0ac216c9e8f None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:35.205745 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ca2387-80da-4eb2-9072-b21d367f0709 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:35.207057 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ca2387-80da-4eb2-9072-b21d367f0709 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:35.207545 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ca2387-80da-4eb2-9072-b21d367f0709 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:35.208101 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ca2387-80da-4eb2-9072-b21d367f0709 None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:35.209211 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ca2387-80da-4eb2-9072-b21d367f0709 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:35.209702 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ca2387-80da-4eb2-9072-b21d367f0709 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:35.211276 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ca2387-80da-4eb2-9072-b21d367f0709 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:35.225614 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 433/861] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:20:35.231155 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ca2387-80da-4eb2-9072-b21d367f0709 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:35.232116 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cbfb4e9-09b2-4eb6-9e7b-91fa4910a5d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:35.232806 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbfb4e9-09b2-4eb6-9e7b-91fa4910a5d6 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:35.233075 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbfb4e9-09b2-4eb6-9e7b-91fa4910a5d6 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:35.233354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbfb4e9-09b2-4eb6-9e7b-91fa4910a5d6 None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:35.234123 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cbfb4e9-09b2-4eb6-9e7b-91fa4910a5d6 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:35.234456 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cbfb4e9-09b2-4eb6-9e7b-91fa4910a5d6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:35.235834 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cbfb4e9-09b2-4eb6-9e7b-91fa4910a5d6 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:35.237387 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:35.241554 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 429/862] 10.222.0.16 () {56 vars in 1126 bytes} [Tue Jul 12 17:20:35 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:35.339894 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:35.350588 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ca2387-80da-4eb2-9072-b21d367f0709 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:35.390450 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c403dd5-c42e-4e80-a2bb-27b6451dd3ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:35.391293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c403dd5-c42e-4e80-a2bb-27b6451dd3ae None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:35.391583 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c403dd5-c42e-4e80-a2bb-27b6451dd3ae None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:35.391866 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c403dd5-c42e-4e80-a2bb-27b6451dd3ae None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:35.400650 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c403dd5-c42e-4e80-a2bb-27b6451dd3ae None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:35.401603 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c403dd5-c42e-4e80-a2bb-27b6451dd3ae None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:35.417256 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-8c403dd5-c42e-4e80-a2bb-27b6451dd3ae None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 12 17:20:35.417256 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:35.417256 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:35.417256 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:35.417256 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:35.417256 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:35.417256 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:35.417256 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:35.417256 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:35.417256 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:35.417256 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:35.417256 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:35.417256 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:20:35.417256 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:20:35.417256 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:20:35.417256 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:20:35.418254 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:35.418254 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:35.418254 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:20:35.418254 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:35.418254 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:20:35.418254 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:35.418254 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:35.418254 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:35.418254 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:35.418254 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:35.418254 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:35.418254 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:35.418254 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:35.418254 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:35.418254 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:35.418254 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:35.418254 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:35.418254 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:35.419318 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:35.419318 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:35.419318 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:20:35.419318 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:20:35.419318 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:20:35.419318 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:20:35.419318 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:20:35.419318 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:20:35.419318 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 12 17:20:35.419318 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:35.419318 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 434/863] 10.222.0.16 () {56 vars in 1117 bytes} [Tue Jul 12 17:20:35 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:35.426092 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cbfb4e9-09b2-4eb6-9e7b-91fa4910a5d6 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:35.534846 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cbfb4e9-09b2-4eb6-9e7b-91fa4910a5d6 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:35.570465 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5259a4-4f0e-406f-b3f5-578d90695225 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:35.570669 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5259a4-4f0e-406f-b3f5-578d90695225 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:35.570780 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5259a4-4f0e-406f-b3f5-578d90695225 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:35.571061 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5259a4-4f0e-406f-b3f5-578d90695225 None admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:35.572341 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc5259a4-4f0e-406f-b3f5-578d90695225 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:35.572341 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc5259a4-4f0e-406f-b3f5-578d90695225 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:35.577890 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc5259a4-4f0e-406f-b3f5-578d90695225 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:35.596424 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 430/864] 10.222.0.16 () {56 vars in 1149 bytes} [Tue Jul 12 17:20:35 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:20:35.602492 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c403dd5-c42e-4e80-a2bb-27b6451dd3ae None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:35.711216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c403dd5-c42e-4e80-a2bb-27b6451dd3ae None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:35.748080 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aff1f24-5ff6-48c7-9c79-3da4975d3075 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:35.748744 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aff1f24-5ff6-48c7-9c79-3da4975d3075 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:35.749032 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aff1f24-5ff6-48c7-9c79-3da4975d3075 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:35.749315 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aff1f24-5ff6-48c7-9c79-3da4975d3075 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:35.772443 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:35.829300 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aff1f24-5ff6-48c7-9c79-3da4975d3075 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:35.831204 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aff1f24-5ff6-48c7-9c79-3da4975d3075 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:35.945486 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 435/865] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:20:35 2022] GET /identity/v3/auth/tokens => generated 1151 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 12 17:20:35.945657 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5259a4-4f0e-406f-b3f5-578d90695225 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:36.005093 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:36.050277 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5259a4-4f0e-406f-b3f5-578d90695225 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:36.088905 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deddfee4-12e8-467d-a537-af6195260715 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:36.089820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deddfee4-12e8-467d-a537-af6195260715 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:36.090015 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deddfee4-12e8-467d-a537-af6195260715 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:36.090270 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deddfee4-12e8-467d-a537-af6195260715 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:36.091437 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19989739-9e96-419b-8bb4-a4bef31b8dc7 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:36.091876 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19989739-9e96-419b-8bb4-a4bef31b8dc7 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:36.092167 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19989739-9e96-419b-8bb4-a4bef31b8dc7 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:36.094044 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 436/866] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:36.100532 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385675ef-6068-4138-82fe-6a7ff790a9e7 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:36.100950 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385675ef-6068-4138-82fe-6a7ff790a9e7 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:36.101121 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deddfee4-12e8-467d-a537-af6195260715 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:36.101861 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385675ef-6068-4138-82fe-6a7ff790a9e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:36.102930 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deddfee4-12e8-467d-a537-af6195260715 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:36.119829 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-385675ef-6068-4138-82fe-6a7ff790a9e7 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:36.121163 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-385675ef-6068-4138-82fe-6a7ff790a9e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:36.121531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-deddfee4-12e8-467d-a537-af6195260715 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:36.121531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:36.121531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:36.121531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:36.121531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:36.121531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:36.121531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:36.121531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:36.121531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:36.121531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:36.121531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:36.121531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:36.121531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:36.121531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:36.121531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:36.123320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:36.123320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:36.123320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:36.123320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:36.123320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:36.123320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:36.123320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:36.123320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:36.123320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:36.123320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:36.123320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:36.123320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:36.123320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:36.123320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:36.123320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:36.123320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:36.123320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:36.124274 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:36.124274 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:36.124274 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:36.124274 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:36.124274 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 431/867] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:36.129154 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deddfee4-12e8-467d-a537-af6195260715 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:36.182169 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:36.206544 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 437/868] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:36 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 12 17:20:36.230093 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385675ef-6068-4138-82fe-6a7ff790a9e7 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:36.265325 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deddfee4-12e8-467d-a537-af6195260715 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:36.275197 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:36.301356 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fef8f7e-bb01-4e6b-95ed-1c6742442e18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:36.302055 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fef8f7e-bb01-4e6b-95ed-1c6742442e18 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:36.302347 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fef8f7e-bb01-4e6b-95ed-1c6742442e18 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:36.302664 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fef8f7e-bb01-4e6b-95ed-1c6742442e18 None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:36.303399 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fef8f7e-bb01-4e6b-95ed-1c6742442e18 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:36.303712 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fef8f7e-bb01-4e6b-95ed-1c6742442e18 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:36.305098 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fef8f7e-bb01-4e6b-95ed-1c6742442e18 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:36.318419 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 432/869] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:36.323577 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fef8f7e-bb01-4e6b-95ed-1c6742442e18 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:36.346463 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385675ef-6068-4138-82fe-6a7ff790a9e7 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:36.384980 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730e548f-9579-4edf-8ece-9e9120a28b94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:36.385796 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730e548f-9579-4edf-8ece-9e9120a28b94 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:36.386083 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730e548f-9579-4edf-8ece-9e9120a28b94 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:36.386366 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730e548f-9579-4edf-8ece-9e9120a28b94 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:36.387075 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-730e548f-9579-4edf-8ece-9e9120a28b94 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:36.388529 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-730e548f-9579-4edf-8ece-9e9120a28b94 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:36.401422 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-730e548f-9579-4edf-8ece-9e9120a28b94 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 12 17:20:36.401422 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:36.401422 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 12 17:20:36.401422 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 12 17:20:36.401422 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 12 17:20:36.401422 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 12 17:20:36.401422 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 12 17:20:36.401422 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:36.401422 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 12 17:20:36.401422 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:36.401422 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 12 17:20:36.401422 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:36.401422 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:36.401422 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:36.401422 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:36.401422 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:36.401422 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:36.402478 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:36.402478 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:36.402478 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:36.402478 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:36.402478 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:36.402478 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:36.402478 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 12 17:20:36.402478 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 12 17:20:36.402478 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 12 17:20:36.402478 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 12 17:20:36.402478 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:36.402478 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:36.402478 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:36.402478 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:36.402478 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:36.402478 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:36.403376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:36.403376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:36.403376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:36.403376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:36.403376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:36.403376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:36.403376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:36.403376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:36.403376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:36.403376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:36.403376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 12 17:20:36.403376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:36.403376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 12 17:20:36.403376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:36.403376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 438/870] 10.222.0.16 () {56 vars in 1126 bytes} [Tue Jul 12 17:20:36 2022] GET /identity/v3/services/object-store => generated 93 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:20:36.409999 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730e548f-9579-4edf-8ece-9e9120a28b94 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:36.432743 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fef8f7e-bb01-4e6b-95ed-1c6742442e18 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:36.458763 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:36.469240 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415a45ac-ae64-4435-9edc-c1fa0e80ccc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:36.470108 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415a45ac-ae64-4435-9edc-c1fa0e80ccc8 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:36.471969 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415a45ac-ae64-4435-9edc-c1fa0e80ccc8 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:36.471969 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415a45ac-ae64-4435-9edc-c1fa0e80ccc8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:36.478607 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415a45ac-ae64-4435-9edc-c1fa0e80ccc8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:36.480183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415a45ac-ae64-4435-9edc-c1fa0e80ccc8 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:36.498327 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-415a45ac-ae64-4435-9edc-c1fa0e80ccc8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:36.498327 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:36.498327 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:36.498327 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:36.498327 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:36.498327 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:36.498327 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:36.498327 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:36.498327 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:36.498327 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:36.498327 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:36.498327 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:36.498327 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:20:36.498327 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:20:36.498327 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:20:36.499297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:20:36.499297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:36.499297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:36.499297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:36.499297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:36.499297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:36.499297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:36.499297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:36.499297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:36.499297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:36.499297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:36.499297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:36.499297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:36.499297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:36.499297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:36.499297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:36.499297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:36.500167 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:20:36.500167 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:20:36.500167 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:20:36.500167 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:20:36.500167 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:20:36.500167 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:20:36.500167 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:36.500167 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:36.500167 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 433/871] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:20:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:20:36.510458 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415a45ac-ae64-4435-9edc-c1fa0e80ccc8 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:36.529307 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730e548f-9579-4edf-8ece-9e9120a28b94 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:36.537646 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:36.568850 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300bf96a-afb1-4fcb-85c5-7eb9b310f1a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:36.569785 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300bf96a-afb1-4fcb-85c5-7eb9b310f1a4 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:36.570228 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300bf96a-afb1-4fcb-85c5-7eb9b310f1a4 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:36.570713 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-300bf96a-afb1-4fcb-85c5-7eb9b310f1a4 None admin] PATH_INFO: `/v3/services` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:36.571592 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300bf96a-afb1-4fcb-85c5-7eb9b310f1a4 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:36.572072 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300bf96a-afb1-4fcb-85c5-7eb9b310f1a4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:36.573974 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-300bf96a-afb1-4fcb-85c5-7eb9b310f1a4 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:36.582063 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 439/872] 10.222.0.16 () {56 vars in 1122 bytes} [Tue Jul 12 17:20:36 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:36.590009 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300bf96a-afb1-4fcb-85c5-7eb9b310f1a4 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:36.633176 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415a45ac-ae64-4435-9edc-c1fa0e80ccc8 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:36.679484 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e85c962-a34a-45dc-9c3a-82ad2a6f1337 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:36.681048 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e85c962-a34a-45dc-9c3a-82ad2a6f1337 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:36.681340 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e85c962-a34a-45dc-9c3a-82ad2a6f1337 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:36.681625 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e85c962-a34a-45dc-9c3a-82ad2a6f1337 None admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:36.682448 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e85c962-a34a-45dc-9c3a-82ad2a6f1337 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:36.683677 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e85c962-a34a-45dc-9c3a-82ad2a6f1337 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:36.685380 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e85c962-a34a-45dc-9c3a-82ad2a6f1337 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:36.706259 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 434/873] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Jul 12 17:20:36 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:36.711937 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e85c962-a34a-45dc-9c3a-82ad2a6f1337 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:36.720537 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-300bf96a-afb1-4fcb-85c5-7eb9b310f1a4 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:36.730050 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:36.758323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ee6163-5a06-497d-bb06-c687ff00d304 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:36.759030 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ee6163-5a06-497d-bb06-c687ff00d304 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:36.759325 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ee6163-5a06-497d-bb06-c687ff00d304 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:36.759609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ee6163-5a06-497d-bb06-c687ff00d304 None admin] PATH_INFO: `/v3/services` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:36.760322 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ee6163-5a06-497d-bb06-c687ff00d304 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:36.760635 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ee6163-5a06-497d-bb06-c687ff00d304 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:36.761999 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ee6163-5a06-497d-bb06-c687ff00d304 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:36.767182 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 440/874] 10.222.0.16 () {56 vars in 1122 bytes} [Tue Jul 12 17:20:36 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:36.768469 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8945639a-52af-4fff-8c81-bfd95ec22be0 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:36.768771 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8945639a-52af-4fff-8c81-bfd95ec22be0 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:36.769060 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8945639a-52af-4fff-8c81-bfd95ec22be0 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:36.772328 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 441/875] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:36.776534 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f37c67d-4e24-4dcc-bdc2-66d008bcc178 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:36.777028 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f37c67d-4e24-4dcc-bdc2-66d008bcc178 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:36.777336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f37c67d-4e24-4dcc-bdc2-66d008bcc178 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:36.800752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-3f37c67d-4e24-4dcc-bdc2-66d008bcc178 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:36.802373 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-3f37c67d-4e24-4dcc-bdc2-66d008bcc178 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:36.834815 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:36.850603 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e85c962-a34a-45dc-9c3a-82ad2a6f1337 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:36.890899 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 442/876] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:36 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:20:36.898084 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a6aae4f-7cb0-4780-aeb9-efc06de6afbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:36.898842 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6aae4f-7cb0-4780-aeb9-efc06de6afbd None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:36.899173 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6aae4f-7cb0-4780-aeb9-efc06de6afbd None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:36.899468 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6aae4f-7cb0-4780-aeb9-efc06de6afbd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:36.900381 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a6aae4f-7cb0-4780-aeb9-efc06de6afbd None admin] RBAC: Adding query filter params (role.id=bef74d4975164f75bcec96ac47afea73, scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2, user.id=3ac0d5d8de4943a3888abee3b24b8a6f) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:36.900703 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a6aae4f-7cb0-4780-aeb9-efc06de6afbd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:36.902228 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a6aae4f-7cb0-4780-aeb9-efc06de6afbd None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:36.909840 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f37c67d-4e24-4dcc-bdc2-66d008bcc178 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:36.910618 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 435/877] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:20:36 2022] GET /identity/v3/role_assignments?user.id=3ac0d5d8de4943a3888abee3b24b8a6f&scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2&role.id=bef74d4975164f75bcec96ac47afea73 => generated 608 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:20:37.028684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f37c67d-4e24-4dcc-bdc2-66d008bcc178 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:37.030810 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:37.069211 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab61ff3a-eb56-44d1-9b78-a51a2060b374 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:37.070205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab61ff3a-eb56-44d1-9b78-a51a2060b374 None admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:37.070710 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab61ff3a-eb56-44d1-9b78-a51a2060b374 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:37.071509 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab61ff3a-eb56-44d1-9b78-a51a2060b374 None admin] PATH_INFO: `/v3/services` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:37.072423 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab61ff3a-eb56-44d1-9b78-a51a2060b374 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:37.073996 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab61ff3a-eb56-44d1-9b78-a51a2060b374 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:37.156271 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 443/878] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Jul 12 17:20:36 2022] POST /identity/v3/services => generated 251 bytes in 247 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:20:37.502152 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c4be4c-44e7-4280-8912-c72cbfeb5362 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:37.502731 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c4be4c-44e7-4280-8912-c72cbfeb5362 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:37.502731 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27c4be4c-44e7-4280-8912-c72cbfeb5362 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:37.504373 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 436/879] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:37.509863 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5b94ac-8b3f-4c0d-a843-8ad79d939cc9 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:37.510330 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5b94ac-8b3f-4c0d-a843-8ad79d939cc9 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:37.510658 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5b94ac-8b3f-4c0d-a843-8ad79d939cc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:37.527903 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-fc5b94ac-8b3f-4c0d-a843-8ad79d939cc9 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:37.529622 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-fc5b94ac-8b3f-4c0d-a843-8ad79d939cc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:37.607592 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 444/880] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:37 2022] POST /identity/v3/auth/tokens => generated 1252 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:20:37.629808 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c4be4c-44e7-4280-8912-c72cbfeb5362 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:37.743530 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27c4be4c-44e7-4280-8912-c72cbfeb5362 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:37.779568 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4d1ee1-677d-413f-b710-93092420f0a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:37.780164 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4d1ee1-677d-413f-b710-93092420f0a5 None admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:37.780451 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4d1ee1-677d-413f-b710-93092420f0a5 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:37.780736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4d1ee1-677d-413f-b710-93092420f0a5 None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:37.781475 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4d1ee1-677d-413f-b710-93092420f0a5 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:37.782885 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4d1ee1-677d-413f-b710-93092420f0a5 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:37.892812 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 437/881] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Jul 12 17:20:37 2022] POST /identity/v3/services => generated 269 bytes in 263 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:20:38.899205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048d1ce4-a6b2-475f-af27-46d72297c080 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:38.899715 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048d1ce4-a6b2-475f-af27-46d72297c080 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:38.899715 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048d1ce4-a6b2-475f-af27-46d72297c080 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:38.901334 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 445/882] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:38.906869 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7615ff1-42d4-48ae-9d2e-883e1702a208 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:38.907307 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7615ff1-42d4-48ae-9d2e-883e1702a208 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:38.907505 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7615ff1-42d4-48ae-9d2e-883e1702a208 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:38.925401 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-a7615ff1-42d4-48ae-9d2e-883e1702a208 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:38.926633 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-a7615ff1-42d4-48ae-9d2e-883e1702a208 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:38.937549 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:39.007258 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 438/883] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:38 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 12 17:20:39.030238 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048d1ce4-a6b2-475f-af27-46d72297c080 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:39.129354 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:39.142896 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048d1ce4-a6b2-475f-af27-46d72297c080 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:39.157604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abce55cc-30d2-47ca-937a-72f5c7eedf97 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:39.157833 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abce55cc-30d2-47ca-937a-72f5c7eedf97 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:39.158242 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abce55cc-30d2-47ca-937a-72f5c7eedf97 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:39.160085 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 439/884] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:39.164899 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfaefdb-69a8-493d-8a68-37a2b48a66e8 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:39.165263 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfaefdb-69a8-493d-8a68-37a2b48a66e8 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:39.165638 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfaefdb-69a8-493d-8a68-37a2b48a66e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:39.182479 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-bbfaefdb-69a8-493d-8a68-37a2b48a66e8 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:39.183555 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d8d105-3241-4bd9-a705-e5bd6096a71e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:39.184925 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-bbfaefdb-69a8-493d-8a68-37a2b48a66e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:39.185026 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d8d105-3241-4bd9-a705-e5bd6096a71e None admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:39.185280 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d8d105-3241-4bd9-a705-e5bd6096a71e None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:39.185561 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d8d105-3241-4bd9-a705-e5bd6096a71e None admin] PATH_INFO: `/v3/services` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:39.186309 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d8d105-3241-4bd9-a705-e5bd6096a71e None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:39.187716 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43d8d105-3241-4bd9-a705-e5bd6096a71e None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:39.254848 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 440/885] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:39 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:20:39.257356 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce534c11-4c3f-46e3-b132-d59f23bbce0f None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:39.257844 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce534c11-4c3f-46e3-b132-d59f23bbce0f None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:39.258397 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce534c11-4c3f-46e3-b132-d59f23bbce0f None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:39.260311 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 441/886] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:39 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:39.265296 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 446/887] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Jul 12 17:20:39 2022] POST /identity/v3/services => generated 241 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:20:39.267889 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e91b70-908b-4ceb-bde5-0951f5b98b4c None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:39.267889 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e91b70-908b-4ceb-bde5-0951f5b98b4c None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:39.267889 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e91b70-908b-4ceb-bde5-0951f5b98b4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:39.280806 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d8d105-3241-4bd9-a705-e5bd6096a71e None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:39.287768 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-77e91b70-908b-4ceb-bde5-0951f5b98b4c None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:39.289388 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-77e91b70-908b-4ceb-bde5-0951f5b98b4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:39.366140 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:39.377934 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 442/888] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:39 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:20:39.405655 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e91b70-908b-4ceb-bde5-0951f5b98b4c None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:39.407157 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d8d105-3241-4bd9-a705-e5bd6096a71e None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:39.450140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8022eae7-8448-4ede-a11d-bd044d5ea820 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:39.451089 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8022eae7-8448-4ede-a11d-bd044d5ea820 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:39.451374 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8022eae7-8448-4ede-a11d-bd044d5ea820 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:39.451652 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8022eae7-8448-4ede-a11d-bd044d5ea820 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:39.465287 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8022eae7-8448-4ede-a11d-bd044d5ea820 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:20:39.474143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-8022eae7-8448-4ede-a11d-bd044d5ea820 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 12 17:20:39.474143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:39.474143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:39.474143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:39.474143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:39.474143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:39.474143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:39.474143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:39.474143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:39.474143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:39.474143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:39.474143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:39.474143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:20:39.474143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:20:39.474143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:20:39.474143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:20:39.475293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:20:39.475293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:20:39.475293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:20:39.475293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:20:39.475293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:39.475293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:39.475293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:39.475293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:39.475293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:39.475293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:39.475293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:39.475293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:39.475293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:39.475293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:39.475293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:39.475293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:39.476462 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:39.476462 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:39.476462 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:39.476462 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:39.476462 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:20:39.476462 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:20:39.476462 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:20:39.476462 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:20:39.476462 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:20:39.476462 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:20:39.476462 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 12 17:20:39.476462 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:39.476462 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 447/889] 10.222.0.16 () {56 vars in 1120 bytes} [Tue Jul 12 17:20:39 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:20:39.483672 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8022eae7-8448-4ede-a11d-bd044d5ea820 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:39.496003 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:39.539175 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e91b70-908b-4ceb-bde5-0951f5b98b4c None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:39.596403 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226ba7f0-d3fc-43bc-944d-c4c764a19348 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:39.598116 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226ba7f0-d3fc-43bc-944d-c4c764a19348 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:39.598970 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226ba7f0-d3fc-43bc-944d-c4c764a19348 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:39.599458 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226ba7f0-d3fc-43bc-944d-c4c764a19348 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:39.600503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-226ba7f0-d3fc-43bc-944d-c4c764a19348 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:39.602981 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-226ba7f0-d3fc-43bc-944d-c4c764a19348 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:39.603947 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8022eae7-8448-4ede-a11d-bd044d5ea820 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:39.618572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-226ba7f0-d3fc-43bc-944d-c4c764a19348 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 12 17:20:39.618572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:39.618572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 12 17:20:39.618572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 12 17:20:39.618572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 12 17:20:39.618572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 12 17:20:39.618572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 12 17:20:39.618572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:39.618572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 12 17:20:39.618572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:39.618572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 12 17:20:39.618572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:39.618572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:39.618572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:39.618572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:39.618572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:39.618572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:39.619604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:39.619604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:39.619604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:39.619604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:39.619604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:39.619604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:39.619604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 12 17:20:39.619604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 12 17:20:39.619604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 12 17:20:39.619604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 12 17:20:39.619604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:39.619604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:39.619604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:39.619604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:39.619604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:39.619604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:39.620443 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:39.620443 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:39.620443 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:39.620443 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:39.620443 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:39.620443 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:39.620443 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:39.620443 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:39.620443 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:39.620443 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:39.620443 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 12 17:20:39.620443 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:39.620443 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 12 17:20:39.620443 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:39.623726 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 443/890] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Jul 12 17:20:39 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:39.629586 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226ba7f0-d3fc-43bc-944d-c4c764a19348 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:39.654364 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6a09ab-492e-4a46-afb5-41dfc2ac839d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:39.655376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6a09ab-492e-4a46-afb5-41dfc2ac839d None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:39.655490 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6a09ab-492e-4a46-afb5-41dfc2ac839d None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:39.655775 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6a09ab-492e-4a46-afb5-41dfc2ac839d None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:39.656548 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6a09ab-492e-4a46-afb5-41dfc2ac839d None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:39.656955 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6a09ab-492e-4a46-afb5-41dfc2ac839d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:39.658331 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6a09ab-492e-4a46-afb5-41dfc2ac839d None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:39.668363 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 448/891] 10.222.0.16 () {56 vars in 1115 bytes} [Tue Jul 12 17:20:39 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:39.674053 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6a09ab-492e-4a46-afb5-41dfc2ac839d None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:39.707878 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:39.757488 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226ba7f0-d3fc-43bc-944d-c4c764a19348 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:39.780756 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:39.794403 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427409e2-dbc6-4557-b406-cc2fc98d4e73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:39.795254 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427409e2-dbc6-4557-b406-cc2fc98d4e73 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:39.795715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427409e2-dbc6-4557-b406-cc2fc98d4e73 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:39.795849 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427409e2-dbc6-4557-b406-cc2fc98d4e73 None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:39.796513 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427409e2-dbc6-4557-b406-cc2fc98d4e73 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:39.796827 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427409e2-dbc6-4557-b406-cc2fc98d4e73 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:39.798235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427409e2-dbc6-4557-b406-cc2fc98d4e73 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:39.803833 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 444/892] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Jul 12 17:20:39 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:39.809554 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427409e2-dbc6-4557-b406-cc2fc98d4e73 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:39.826059 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6a09ab-492e-4a46-afb5-41dfc2ac839d None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:39.869447 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e577da-0d02-43ca-9856-493aa6a50a38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:39.870920 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e577da-0d02-43ca-9856-493aa6a50a38 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:39.871213 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e577da-0d02-43ca-9856-493aa6a50a38 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:39.872012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e577da-0d02-43ca-9856-493aa6a50a38 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:39.885699 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e577da-0d02-43ca-9856-493aa6a50a38 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:39.888128 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e577da-0d02-43ca-9856-493aa6a50a38 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:39.907438 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-a1e577da-0d02-43ca-9856-493aa6a50a38 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:39.907438 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:39.907438 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:39.907438 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:39.907438 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:39.907438 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:39.907438 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:39.907438 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:39.907438 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:39.907438 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:39.907438 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:39.907438 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:39.907438 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:39.907438 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:39.907438 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:39.908807 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:39.908807 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:39.908807 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:39.908807 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:39.908807 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:39.908807 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:39.908807 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:39.908807 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:39.908807 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:39.908807 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:39.908807 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:39.908807 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:39.908807 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:39.908807 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:39.908807 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:39.908807 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:39.908807 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:39.909692 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:39.909692 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:39.909692 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:39.909692 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:39.909692 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 449/893] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:39.916456 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e577da-0d02-43ca-9856-493aa6a50a38 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:39.935652 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427409e2-dbc6-4557-b406-cc2fc98d4e73 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:39.960143 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:39.973046 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae43d04-a20c-47a0-8760-885ff60b5ee4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:39.974056 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae43d04-a20c-47a0-8760-885ff60b5ee4 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:39.974573 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae43d04-a20c-47a0-8760-885ff60b5ee4 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:39.975051 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae43d04-a20c-47a0-8760-885ff60b5ee4 None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:39.976261 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae43d04-a20c-47a0-8760-885ff60b5ee4 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:39.976767 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae43d04-a20c-47a0-8760-885ff60b5ee4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:39.978306 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae43d04-a20c-47a0-8760-885ff60b5ee4 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:39.988956 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 445/894] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Jul 12 17:20:39 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:39.990206 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae43d04-a20c-47a0-8760-885ff60b5ee4 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:40.056857 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e577da-0d02-43ca-9856-493aa6a50a38 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:40.082599 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:40.094341 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2734d99-3280-4cc2-b13f-f37eea0edc9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:40.095042 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2734d99-3280-4cc2-b13f-f37eea0edc9c None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:40.095335 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2734d99-3280-4cc2-b13f-f37eea0edc9c None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:40.095621 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2734d99-3280-4cc2-b13f-f37eea0edc9c None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:40.096357 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2734d99-3280-4cc2-b13f-f37eea0edc9c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:40.096677 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2734d99-3280-4cc2-b13f-f37eea0edc9c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:40.098058 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2734d99-3280-4cc2-b13f-f37eea0edc9c None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:40.105199 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae43d04-a20c-47a0-8760-885ff60b5ee4 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:40.111092 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 450/895] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:40.112381 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae34dc39-4f82-4643-b4c0-842c873bf41a None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:40.112675 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae34dc39-4f82-4643-b4c0-842c873bf41a None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:40.112958 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae34dc39-4f82-4643-b4c0-842c873bf41a None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:40.118092 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 451/896] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:40 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:40.119000 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae34dc39-4f82-4643-b4c0-842c873bf41a None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:40.148534 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634f1cbb-277e-4932-b476-25b0ee4e3d1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:40.149513 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634f1cbb-277e-4932-b476-25b0ee4e3d1b None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:40.149951 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634f1cbb-277e-4932-b476-25b0ee4e3d1b None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:40.150383 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634f1cbb-277e-4932-b476-25b0ee4e3d1b None admin] PATH_INFO: `/v3/endpoints` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:40.151458 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634f1cbb-277e-4932-b476-25b0ee4e3d1b None admin] RBAC: Adding query filter params (interface=public, service_id=2d5da9cf11fc481fa151e5d751692b38, region_id=RegionOne) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:40.151936 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634f1cbb-277e-4932-b476-25b0ee4e3d1b None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:40.153530 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634f1cbb-277e-4932-b476-25b0ee4e3d1b None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:40.160040 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 446/897] 10.222.0.16 () {56 vars in 1251 bytes} [Tue Jul 12 17:20:39 2022] GET /identity/v3/endpoints?service_id=2d5da9cf11fc481fa151e5d751692b38&interface=public®ion_id=RegionOne => generated 194 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:40.161642 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25544070-f146-40ca-9181-172a57aedfcb None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:40.162084 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25544070-f146-40ca-9181-172a57aedfcb None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:40.162371 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25544070-f146-40ca-9181-172a57aedfcb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:40.187375 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-25544070-f146-40ca-9181-172a57aedfcb None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:40.189256 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-25544070-f146-40ca-9181-172a57aedfcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:40.259624 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae34dc39-4f82-4643-b4c0-842c873bf41a None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:40.260205 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:40.288823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 447/898] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:40 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:20:40.289966 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25544070-f146-40ca-9181-172a57aedfcb None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:40.316071 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6418dd7f-6f1a-4166-ae9a-06ac2c877b24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:40.317206 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6418dd7f-6f1a-4166-ae9a-06ac2c877b24 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:40.317769 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6418dd7f-6f1a-4166-ae9a-06ac2c877b24 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:40.318056 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6418dd7f-6f1a-4166-ae9a-06ac2c877b24 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:40.329192 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6418dd7f-6f1a-4166-ae9a-06ac2c877b24 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:40.330752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6418dd7f-6f1a-4166-ae9a-06ac2c877b24 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:40.346752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-6418dd7f-6f1a-4166-ae9a-06ac2c877b24 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 12 17:20:40.346752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:40.346752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:40.346752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:40.346752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:40.346752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:40.346752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:40.346752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:40.346752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:40.346752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:40.346752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:40.346752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:40.346752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:20:40.346752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:20:40.346752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:20:40.346752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:20:40.347794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:40.347794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:40.347794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:20:40.347794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:40.347794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:20:40.347794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:40.347794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:40.347794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:40.347794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:40.347794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:40.347794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:40.347794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:40.347794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:40.347794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:40.347794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:40.347794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:40.347794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:40.347794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:40.348775 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:40.348775 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:40.348775 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:20:40.348775 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:20:40.348775 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:20:40.348775 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:20:40.348775 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:20:40.348775 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:20:40.348775 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 12 17:20:40.348775 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:40.348775 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 452/899] 10.222.0.16 () {56 vars in 1117 bytes} [Tue Jul 12 17:20:40 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:40.348775 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6418dd7f-6f1a-4166-ae9a-06ac2c877b24 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:40.397170 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:40.402701 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25544070-f146-40ca-9181-172a57aedfcb None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:40.441747 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce18242-387d-480c-92cf-22c97673923c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:40.442420 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce18242-387d-480c-92cf-22c97673923c None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:40.442749 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce18242-387d-480c-92cf-22c97673923c None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:40.443030 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce18242-387d-480c-92cf-22c97673923c None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:40.443716 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce18242-387d-480c-92cf-22c97673923c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:40.445091 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce18242-387d-480c-92cf-22c97673923c None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:40.451946 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 448/900] 10.222.0.16 () {56 vars in 1087 bytes} [Tue Jul 12 17:20:40 2022] GET /identity/v3/services => generated 1497 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:20:40.453032 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce18242-387d-480c-92cf-22c97673923c None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:40.474178 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6418dd7f-6f1a-4166-ae9a-06ac2c877b24 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:40.510680 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:40.524549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42e3bfd-d0fd-40e2-915f-5378c5e04131 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:40.525420 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42e3bfd-d0fd-40e2-915f-5378c5e04131 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:40.525895 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42e3bfd-d0fd-40e2-915f-5378c5e04131 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:40.526112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42e3bfd-d0fd-40e2-915f-5378c5e04131 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:40.527144 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42e3bfd-d0fd-40e2-915f-5378c5e04131 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:40.529483 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42e3bfd-d0fd-40e2-915f-5378c5e04131 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:40.549619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-f42e3bfd-d0fd-40e2-915f-5378c5e04131 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 12 17:20:40.549619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:40.549619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 12 17:20:40.549619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 12 17:20:40.549619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 12 17:20:40.549619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 12 17:20:40.549619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 12 17:20:40.549619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:40.549619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 12 17:20:40.549619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:40.549619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 12 17:20:40.549619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:40.549619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:40.549619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:40.549619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:40.549619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:40.550782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:40.550782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:40.550782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:40.550782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:40.550782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:40.550782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:40.550782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:40.550782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 12 17:20:40.550782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 12 17:20:40.550782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 12 17:20:40.550782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 12 17:20:40.550782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:40.550782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:40.550782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:40.550782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:40.550782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:40.551637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:40.551637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:40.551637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:40.551637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:40.551637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:40.551637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:40.551637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:40.551637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:40.551637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:40.551637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:40.551637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:40.551637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 12 17:20:40.551637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:40.551637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 12 17:20:40.551637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:40.554827 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 453/901] 10.222.0.16 () {56 vars in 1132 bytes} [Tue Jul 12 17:20:40 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:40.557493 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42e3bfd-d0fd-40e2-915f-5378c5e04131 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:40.587986 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce18242-387d-480c-92cf-22c97673923c None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:40.631153 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c12f35-7308-4a88-be93-ef6416aa8f49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:40.632259 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c12f35-7308-4a88-be93-ef6416aa8f49 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:40.632763 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c12f35-7308-4a88-be93-ef6416aa8f49 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:40.633051 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c12f35-7308-4a88-be93-ef6416aa8f49 None admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:40.636137 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c12f35-7308-4a88-be93-ef6416aa8f49 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:40.636137 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c12f35-7308-4a88-be93-ef6416aa8f49 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:40.636137 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c12f35-7308-4a88-be93-ef6416aa8f49 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:40.654493 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 449/902] 10.222.0.16 () {56 vars in 1149 bytes} [Tue Jul 12 17:20:40 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:40.660227 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c12f35-7308-4a88-be93-ef6416aa8f49 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:40.706163 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42e3bfd-d0fd-40e2-915f-5378c5e04131 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:40.718264 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:40.750164 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb8af3d-1d63-486c-858b-22a45fdf756c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:40.751155 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb8af3d-1d63-486c-858b-22a45fdf756c None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:40.751454 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb8af3d-1d63-486c-858b-22a45fdf756c None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:40.751742 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb8af3d-1d63-486c-858b-22a45fdf756c None admin] PATH_INFO: `/v3/services` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:40.752465 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb8af3d-1d63-486c-858b-22a45fdf756c None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:40.752843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb8af3d-1d63-486c-858b-22a45fdf756c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:40.755007 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb8af3d-1d63-486c-858b-22a45fdf756c None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:40.761671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 454/903] 10.222.0.16 () {56 vars in 1126 bytes} [Tue Jul 12 17:20:40 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:40.767231 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb8af3d-1d63-486c-858b-22a45fdf756c None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:40.775832 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c12f35-7308-4a88-be93-ef6416aa8f49 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:40.778626 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:40.814563 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec7809c-f0d3-47be-bc6e-37ef1d99a563 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:40.815751 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec7809c-f0d3-47be-bc6e-37ef1d99a563 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:40.816264 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec7809c-f0d3-47be-bc6e-37ef1d99a563 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:40.816749 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec7809c-f0d3-47be-bc6e-37ef1d99a563 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:40.888627 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb8af3d-1d63-486c-858b-22a45fdf756c None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:40.893222 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec7809c-f0d3-47be-bc6e-37ef1d99a563 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:40.894841 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec7809c-f0d3-47be-bc6e-37ef1d99a563 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:40.926216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b4075b-e995-46cc-823f-1b93e5aadc63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:40.927049 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b4075b-e995-46cc-823f-1b93e5aadc63 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:40.927556 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b4075b-e995-46cc-823f-1b93e5aadc63 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:40.927850 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b4075b-e995-46cc-823f-1b93e5aadc63 None admin] PATH_INFO: `/v3/services` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:40.928576 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b4075b-e995-46cc-823f-1b93e5aadc63 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:40.928889 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b4075b-e995-46cc-823f-1b93e5aadc63 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:40.930385 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b4075b-e995-46cc-823f-1b93e5aadc63 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:40.936334 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 455/904] 10.222.0.16 () {56 vars in 1126 bytes} [Tue Jul 12 17:20:40 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:20:40.942281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b4075b-e995-46cc-823f-1b93e5aadc63 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:40.981462 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:41.018916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 450/905] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:20:40 2022] GET /identity/v3/auth/tokens => generated 1464 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 12 17:20:41.021641 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec7809c-f0d3-47be-bc6e-37ef1d99a563 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:41.042805 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:41.062642 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b4075b-e995-46cc-823f-1b93e5aadc63 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:41.110513 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2b494f-9035-4580-8379-abc5bdfe0dca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:41.111618 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2b494f-9035-4580-8379-abc5bdfe0dca None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:41.112102 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2b494f-9035-4580-8379-abc5bdfe0dca None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:41.112575 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2b494f-9035-4580-8379-abc5bdfe0dca None admin] PATH_INFO: `/v3/endpoints` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:41.113601 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2b494f-9035-4580-8379-abc5bdfe0dca None admin] RBAC: Adding query filter params (interface=public, service_id=914587a10da54c648dcd7ee4dad22dbc, region_id=RegionOne) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:41.114114 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2b494f-9035-4580-8379-abc5bdfe0dca None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:41.115720 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2b494f-9035-4580-8379-abc5bdfe0dca None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:41.129474 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 456/906] 10.222.0.16 () {56 vars in 1251 bytes} [Tue Jul 12 17:20:40 2022] GET /identity/v3/endpoints?service_id=914587a10da54c648dcd7ee4dad22dbc&interface=public®ion_id=RegionOne => generated 194 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:41.130750 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2b494f-9035-4580-8379-abc5bdfe0dca None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:41.144281 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec7809c-f0d3-47be-bc6e-37ef1d99a563 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:41.181741 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9bc6827-454f-4d57-90e2-beba8c7c479e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:41.182655 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9bc6827-454f-4d57-90e2-beba8c7c479e None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:41.183216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9bc6827-454f-4d57-90e2-beba8c7c479e None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:41.183417 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9bc6827-454f-4d57-90e2-beba8c7c479e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:41.196621 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9bc6827-454f-4d57-90e2-beba8c7c479e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:41.198259 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9bc6827-454f-4d57-90e2-beba8c7c479e None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:41.218331 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-c9bc6827-454f-4d57-90e2-beba8c7c479e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:41.218331 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:41.218331 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:41.218331 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:41.218331 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:41.218331 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:41.218331 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:41.218331 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:41.218331 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:41.218331 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:41.218331 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:41.218331 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:41.218331 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:41.218331 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:41.218331 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:41.219299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:41.219299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:41.219299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:41.219299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:41.219299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:41.219299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:41.219299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:41.219299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:41.219299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:41.219299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:41.219299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:41.219299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:41.219299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:41.219299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:41.219299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:41.219299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:41.219299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:41.220195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:41.220195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:41.220195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:41.220195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:41.224424 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 451/907] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:20:41.225803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9bc6827-454f-4d57-90e2-beba8c7c479e None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:41.242336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:41.246865 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2b494f-9035-4580-8379-abc5bdfe0dca None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:41.284227 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cdf826b-7150-4d55-bb3f-7743c459e6c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:41.285352 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdf826b-7150-4d55-bb3f-7743c459e6c8 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:41.285837 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdf826b-7150-4d55-bb3f-7743c459e6c8 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:41.286323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdf826b-7150-4d55-bb3f-7743c459e6c8 None admin] PATH_INFO: `/v3/services` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:41.287304 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cdf826b-7150-4d55-bb3f-7743c459e6c8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:41.288721 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:41.290209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cdf826b-7150-4d55-bb3f-7743c459e6c8 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:41.304632 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 457/908] 10.222.0.16 () {56 vars in 1087 bytes} [Tue Jul 12 17:20:41 2022] GET /identity/v3/services => generated 1497 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:20:41.345686 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9bc6827-454f-4d57-90e2-beba8c7c479e None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:41.385638 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0898b2af-e636-4847-8f24-f77827e2b24d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:41.386317 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0898b2af-e636-4847-8f24-f77827e2b24d None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:41.386701 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0898b2af-e636-4847-8f24-f77827e2b24d None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:41.387088 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0898b2af-e636-4847-8f24-f77827e2b24d None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:41.388346 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0898b2af-e636-4847-8f24-f77827e2b24d None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:41.388659 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0898b2af-e636-4847-8f24-f77827e2b24d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:41.390063 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0898b2af-e636-4847-8f24-f77827e2b24d None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:41.405399 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 452/909] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:41.411091 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cdf826b-7150-4d55-bb3f-7743c459e6c8 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:41.452894 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157daa02-9d6d-4a37-8354-f5588b5fb632 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:41.453211 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157daa02-9d6d-4a37-8354-f5588b5fb632 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:41.453462 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157daa02-9d6d-4a37-8354-f5588b5fb632 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:41.456293 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 453/910] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 12 17:20:41.460785 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa27c8eb-0693-478c-8bed-9308e48b83af None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:41.461081 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa27c8eb-0693-478c-8bed-9308e48b83af None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:41.461371 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa27c8eb-0693-478c-8bed-9308e48b83af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:41.480301 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-fa27c8eb-0693-478c-8bed-9308e48b83af None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:41.481550 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-fa27c8eb-0693-478c-8bed-9308e48b83af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:41.540083 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cdf826b-7150-4d55-bb3f-7743c459e6c8 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:41.563398 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 454/911] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:41 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:20:41.586472 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c898d84-2cff-4b2c-99ad-0acf7de3f102 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:41.586472 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c898d84-2cff-4b2c-99ad-0acf7de3f102 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:41.586472 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c898d84-2cff-4b2c-99ad-0acf7de3f102 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:41.586472 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c898d84-2cff-4b2c-99ad-0acf7de3f102 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:41.586846 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa27c8eb-0693-478c-8bed-9308e48b83af None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:41.591645 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c898d84-2cff-4b2c-99ad-0acf7de3f102 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:41.593114 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c898d84-2cff-4b2c-99ad-0acf7de3f102 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:41.617770 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-8c898d84-2cff-4b2c-99ad-0acf7de3f102 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:41.617770 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:41.617770 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:41.617770 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:41.617770 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:41.617770 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:41.617770 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:41.617770 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:41.617770 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:41.617770 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:41.617770 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:41.617770 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:41.617770 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:20:41.617770 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:20:41.617770 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:20:41.618747 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:20:41.618747 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:41.618747 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:41.618747 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:41.618747 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:41.618747 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:41.618747 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:41.618747 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:41.618747 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:41.618747 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:41.618747 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:41.618747 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:41.618747 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:41.618747 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:41.618747 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:41.618747 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:41.618747 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:41.619730 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:20:41.619730 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:20:41.619730 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:20:41.619730 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:20:41.619730 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:20:41.619730 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:20:41.619730 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:41.619730 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:41.621412 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 458/912] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:20:41 2022] GET /identity/v3/projects/service => generated 88 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:41.626614 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c898d84-2cff-4b2c-99ad-0acf7de3f102 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:41.642773 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:41.663617 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:41.725547 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa27c8eb-0693-478c-8bed-9308e48b83af None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:41.751131 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c898d84-2cff-4b2c-99ad-0acf7de3f102 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:41.772167 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a57aeb8-fb8e-4e19-b47f-9766b2297340 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:41.773865 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a57aeb8-fb8e-4e19-b47f-9766b2297340 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:41.774648 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a57aeb8-fb8e-4e19-b47f-9766b2297340 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:41.775132 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a57aeb8-fb8e-4e19-b47f-9766b2297340 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:41.776260 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a57aeb8-fb8e-4e19-b47f-9766b2297340 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:41.778671 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a57aeb8-fb8e-4e19-b47f-9766b2297340 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:41.788046 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9b4ec9-703d-4ee0-b586-118ff60b1451 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:41.788716 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9b4ec9-703d-4ee0-b586-118ff60b1451 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:41.789005 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9b4ec9-703d-4ee0-b586-118ff60b1451 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:41.789287 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9b4ec9-703d-4ee0-b586-118ff60b1451 None admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:41.790080 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d9b4ec9-703d-4ee0-b586-118ff60b1451 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:41.790463 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d9b4ec9-703d-4ee0-b586-118ff60b1451 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:41.791972 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d9b4ec9-703d-4ee0-b586-118ff60b1451 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:41.796326 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-6a57aeb8-fb8e-4e19-b47f-9766b2297340 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 12 17:20:41.796326 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:41.796326 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 12 17:20:41.796326 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 12 17:20:41.796326 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 12 17:20:41.796326 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 12 17:20:41.796326 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 12 17:20:41.796326 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:41.796326 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 12 17:20:41.796326 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:41.796326 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 12 17:20:41.796326 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:41.796326 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:41.796326 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:41.796326 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:41.796326 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:41.796326 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:41.797293 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:41.797293 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:41.797293 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:41.797293 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:41.797293 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:41.797293 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:41.797293 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 12 17:20:41.797293 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 12 17:20:41.797293 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 12 17:20:41.797293 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 12 17:20:41.797293 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:41.797293 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:41.797293 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:41.797293 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:41.797293 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:41.797293 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:41.798149 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:41.798149 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:41.798149 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:41.798149 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:41.798149 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:41.798149 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:41.798149 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:41.798149 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:41.798149 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:41.798149 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:41.798149 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 12 17:20:41.798149 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:41.798149 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 12 17:20:41.798149 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:41.801580 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 455/913] 10.222.0.16 () {56 vars in 1126 bytes} [Tue Jul 12 17:20:41 2022] GET /identity/v3/services/object-store => generated 93 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:20:41.806714 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 459/914] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Jul 12 17:20:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:20:41.807786 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a57aeb8-fb8e-4e19-b47f-9766b2297340 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:41.812680 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9b4ec9-703d-4ee0-b586-118ff60b1451 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:41.921033 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:41.929634 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a57aeb8-fb8e-4e19-b47f-9766b2297340 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:41.937909 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9b4ec9-703d-4ee0-b586-118ff60b1451 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:41.950823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:41.973987 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265422c7-878f-45fa-9bc2-b0edeb46ac3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:41.974667 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265422c7-878f-45fa-9bc2-b0edeb46ac3c None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:41.974961 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265422c7-878f-45fa-9bc2-b0edeb46ac3c None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:41.975242 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265422c7-878f-45fa-9bc2-b0edeb46ac3c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:41.976337 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-265422c7-878f-45fa-9bc2-b0edeb46ac3c None admin] RBAC: Adding query filter params (role.id=feb6a96f7b6c4f0a8d64be226f85ee6c, scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2, user.id=3ac0d5d8de4943a3888abee3b24b8a6f) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:41.976680 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-265422c7-878f-45fa-9bc2-b0edeb46ac3c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:41.977596 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e47a8d-2d88-438d-a576-fedf3736da27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:41.978220 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-265422c7-878f-45fa-9bc2-b0edeb46ac3c None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:41.978742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e47a8d-2d88-438d-a576-fedf3736da27 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:41.979225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e47a8d-2d88-438d-a576-fedf3736da27 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:41.979710 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e47a8d-2d88-438d-a576-fedf3736da27 None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:41.982020 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28e47a8d-2d88-438d-a576-fedf3736da27 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:41.982599 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28e47a8d-2d88-438d-a576-fedf3736da27 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:41.984686 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28e47a8d-2d88-438d-a576-fedf3736da27 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:41.991222 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 456/915] 10.222.0.16 () {56 vars in 1122 bytes} [Tue Jul 12 17:20:41 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:41.991349 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 460/916] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:20:41 2022] GET /identity/v3/role_assignments?user.id=3ac0d5d8de4943a3888abee3b24b8a6f&scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2&role.id=feb6a96f7b6c4f0a8d64be226f85ee6c => generated 259 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:20:42.002524 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e47a8d-2d88-438d-a576-fedf3736da27 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:42.125641 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e47a8d-2d88-438d-a576-fedf3736da27 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:42.170122 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbda34fd-f7ec-4937-9cd5-5a7cdb4e7856 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:42.171126 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbda34fd-f7ec-4937-9cd5-5a7cdb4e7856 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:42.171597 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbda34fd-f7ec-4937-9cd5-5a7cdb4e7856 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:42.172079 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbda34fd-f7ec-4937-9cd5-5a7cdb4e7856 None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:42.173075 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbda34fd-f7ec-4937-9cd5-5a7cdb4e7856 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:42.173566 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbda34fd-f7ec-4937-9cd5-5a7cdb4e7856 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:42.175128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbda34fd-f7ec-4937-9cd5-5a7cdb4e7856 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:42.186545 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 457/917] 10.222.0.16 () {56 vars in 1122 bytes} [Tue Jul 12 17:20:41 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:42.188431 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265422c7-878f-45fa-9bc2-b0edeb46ac3c None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:42.301546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265422c7-878f-45fa-9bc2-b0edeb46ac3c None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:42.338980 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33515a0-80a3-4438-81bb-e5b30f42b52c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:42.339648 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33515a0-80a3-4438-81bb-e5b30f42b52c None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:42.339965 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33515a0-80a3-4438-81bb-e5b30f42b52c None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:42.340278 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33515a0-80a3-4438-81bb-e5b30f42b52c None admin] PATH_INFO: `/v3/endpoints` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:42.341214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c33515a0-80a3-4438-81bb-e5b30f42b52c None admin] RBAC: Adding query filter params (interface=public, service_id=2b75d6e9520b406e8298d5868e24fc50, region_id=RegionOne) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:42.341533 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c33515a0-80a3-4438-81bb-e5b30f42b52c None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:42.343009 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c33515a0-80a3-4438-81bb-e5b30f42b52c None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:42.349131 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 461/918] 10.222.0.16 () {56 vars in 1251 bytes} [Tue Jul 12 17:20:42 2022] GET /identity/v3/endpoints?service_id=2b75d6e9520b406e8298d5868e24fc50&interface=public®ion_id=RegionOne => generated 194 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:42.354976 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbda34fd-f7ec-4937-9cd5-5a7cdb4e7856 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:42.396398 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:42.473031 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbda34fd-f7ec-4937-9cd5-5a7cdb4e7856 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:42.511586 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a97d1b4-4ed6-4b30-9fcf-d79c5ebdddc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:42.512621 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a97d1b4-4ed6-4b30-9fcf-d79c5ebdddc8 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:42.512908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a97d1b4-4ed6-4b30-9fcf-d79c5ebdddc8 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:42.513191 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a97d1b4-4ed6-4b30-9fcf-d79c5ebdddc8 None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:42.513886 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a97d1b4-4ed6-4b30-9fcf-d79c5ebdddc8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:42.515470 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a97d1b4-4ed6-4b30-9fcf-d79c5ebdddc8 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:42.522207 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 458/919] 10.222.0.16 () {56 vars in 1087 bytes} [Tue Jul 12 17:20:42 2022] GET /identity/v3/services => generated 1497 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:20:42.625431 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde7ac05-5886-453b-943f-96f660d07405 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:42.625770 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde7ac05-5886-453b-943f-96f660d07405 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:42.625999 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde7ac05-5886-453b-943f-96f660d07405 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:42.627726 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 462/920] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:42.633396 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa1666c-ac3f-4a10-adfd-962b04688a91 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:42.633703 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa1666c-ac3f-4a10-adfd-962b04688a91 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:42.633978 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa1666c-ac3f-4a10-adfd-962b04688a91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:42.654289 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-7fa1666c-ac3f-4a10-adfd-962b04688a91 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:42.655585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-7fa1666c-ac3f-4a10-adfd-962b04688a91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:42.732993 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 459/921] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:42 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:20:42.755789 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde7ac05-5886-453b-943f-96f660d07405 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:42.769012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:42.866981 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde7ac05-5886-453b-943f-96f660d07405 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:42.904493 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b529bd11-9ba1-42ae-b099-d444acbf9255 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:42.905211 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b529bd11-9ba1-42ae-b099-d444acbf9255 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:42.905497 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b529bd11-9ba1-42ae-b099-d444acbf9255 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:42.905770 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b529bd11-9ba1-42ae-b099-d444acbf9255 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:42.906487 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b529bd11-9ba1-42ae-b099-d444acbf9255 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:42.907959 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b529bd11-9ba1-42ae-b099-d444acbf9255 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:42.920819 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-b529bd11-9ba1-42ae-b099-d444acbf9255 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 12 17:20:42.920819 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:42.920819 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 12 17:20:42.920819 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 12 17:20:42.920819 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 12 17:20:42.920819 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 12 17:20:42.920819 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 12 17:20:42.920819 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:42.920819 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 12 17:20:42.920819 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:42.920819 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 12 17:20:42.920819 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:42.920819 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:42.920819 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:42.920819 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:42.920819 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:42.920819 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:42.921848 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:42.921848 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:42.921848 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:42.921848 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:42.921848 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:42.921848 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:42.921848 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 12 17:20:42.921848 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 12 17:20:42.921848 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 12 17:20:42.921848 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 12 17:20:42.921848 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:42.921848 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:42.921848 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:42.921848 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:42.921848 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:42.921848 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:42.922734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:42.922734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:42.922734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:42.922734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:42.922734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:42.922734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:42.922734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:42.922734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:42.922734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:42.922734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:42.922734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 12 17:20:42.922734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:42.922734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 12 17:20:42.922734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:42.922734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 463/922] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Jul 12 17:20:42 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:42.928663 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa1666c-ac3f-4a10-adfd-962b04688a91 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:42.962795 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:43.035132 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa1666c-ac3f-4a10-adfd-962b04688a91 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:43.071000 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c14d831-5f31-433e-a4a5-86f54c01e4ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:43.071673 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c14d831-5f31-433e-a4a5-86f54c01e4ae None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:43.071956 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c14d831-5f31-433e-a4a5-86f54c01e4ae None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:43.072232 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c14d831-5f31-433e-a4a5-86f54c01e4ae None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:43.073022 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c14d831-5f31-433e-a4a5-86f54c01e4ae None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:43.073332 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c14d831-5f31-433e-a4a5-86f54c01e4ae None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:43.075485 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c14d831-5f31-433e-a4a5-86f54c01e4ae None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:43.081180 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 460/923] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Jul 12 17:20:42 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:43.087497 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b529bd11-9ba1-42ae-b099-d444acbf9255 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:43.173821 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:43.201417 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b529bd11-9ba1-42ae-b099-d444acbf9255 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:43.238909 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8736425b-9b87-4afc-b75d-43c8781aa410 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:43.239589 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8736425b-9b87-4afc-b75d-43c8781aa410 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:43.239882 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8736425b-9b87-4afc-b75d-43c8781aa410 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:43.240166 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8736425b-9b87-4afc-b75d-43c8781aa410 None admin] PATH_INFO: `/v3/services` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:43.240890 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8736425b-9b87-4afc-b75d-43c8781aa410 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:43.241203 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8736425b-9b87-4afc-b75d-43c8781aa410 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:43.242619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8736425b-9b87-4afc-b75d-43c8781aa410 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:43.248732 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 464/924] 10.222.0.16 () {56 vars in 1114 bytes} [Tue Jul 12 17:20:43 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:43.254280 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c14d831-5f31-433e-a4a5-86f54c01e4ae None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:43.351877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:43.359783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c14d831-5f31-433e-a4a5-86f54c01e4ae None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:43.400672 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f3e09d-c7c2-4108-b61c-ac110b8955c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:43.401340 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f3e09d-c7c2-4108-b61c-ac110b8955c9 None admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:43.401631 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f3e09d-c7c2-4108-b61c-ac110b8955c9 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:43.401915 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f3e09d-c7c2-4108-b61c-ac110b8955c9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:43.402731 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f3e09d-c7c2-4108-b61c-ac110b8955c9 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:43.404108 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f3e09d-c7c2-4108-b61c-ac110b8955c9 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:43.484871 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 461/925] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Jul 12 17:20:43 2022] POST /identity/v3/endpoints => generated 346 bytes in 231 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:20:43.581619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f79083-6a10-4b06-942a-1bffb4ecaa48 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:43.581917 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f79083-6a10-4b06-942a-1bffb4ecaa48 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:43.582207 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f79083-6a10-4b06-942a-1bffb4ecaa48 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:43.583901 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 465/926] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:43.589546 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f639bda-73c0-4e11-9ebf-614605ad719e None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:43.590377 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f639bda-73c0-4e11-9ebf-614605ad719e None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:43.590666 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f639bda-73c0-4e11-9ebf-614605ad719e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:43.609372 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-4f639bda-73c0-4e11-9ebf-614605ad719e None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:43.610631 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-4f639bda-73c0-4e11-9ebf-614605ad719e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:43.691500 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 462/927] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:43 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 12 17:20:43.716019 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f79083-6a10-4b06-942a-1bffb4ecaa48 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:43.828176 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f79083-6a10-4b06-942a-1bffb4ecaa48 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:43.865100 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6463a341-3f5a-4678-bb9b-03de49fd13f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:43.866314 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6463a341-3f5a-4678-bb9b-03de49fd13f2 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:43.866851 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6463a341-3f5a-4678-bb9b-03de49fd13f2 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:43.867345 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6463a341-3f5a-4678-bb9b-03de49fd13f2 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:43.868202 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6463a341-3f5a-4678-bb9b-03de49fd13f2 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:43.869820 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6463a341-3f5a-4678-bb9b-03de49fd13f2 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:43.883658 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-6463a341-3f5a-4678-bb9b-03de49fd13f2 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 12 17:20:43.883658 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:43.883658 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 12 17:20:43.883658 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 12 17:20:43.883658 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 12 17:20:43.883658 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 12 17:20:43.883658 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 12 17:20:43.883658 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:43.883658 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 12 17:20:43.883658 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:43.883658 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 12 17:20:43.883658 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:43.883658 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:43.883658 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:43.883658 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:43.883658 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:43.884684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:43.884684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:43.884684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:43.884684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:43.884684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:43.884684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:43.884684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:43.884684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 12 17:20:43.884684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 12 17:20:43.884684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 12 17:20:43.884684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 12 17:20:43.884684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:43.884684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:43.884684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:43.884684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:43.884684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:43.885538 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:43.885538 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:43.885538 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:43.885538 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:43.885538 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:43.885538 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:43.885538 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:43.885538 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:43.885538 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:43.885538 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:43.885538 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:43.885538 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 12 17:20:43.885538 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:43.885538 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 12 17:20:43.885538 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:43.888522 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 466/928] 10.222.0.16 () {56 vars in 1132 bytes} [Tue Jul 12 17:20:43 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:43.894765 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f639bda-73c0-4e11-9ebf-614605ad719e None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:43.977988 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:44.001965 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f639bda-73c0-4e11-9ebf-614605ad719e None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:44.040317 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42f8202-7ed2-4f43-ad7c-c4597893a251 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:44.040996 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42f8202-7ed2-4f43-ad7c-c4597893a251 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:44.041318 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42f8202-7ed2-4f43-ad7c-c4597893a251 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:44.041601 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42f8202-7ed2-4f43-ad7c-c4597893a251 None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:44.042318 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b42f8202-7ed2-4f43-ad7c-c4597893a251 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:44.042738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b42f8202-7ed2-4f43-ad7c-c4597893a251 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:44.044663 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b42f8202-7ed2-4f43-ad7c-c4597893a251 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:44.050027 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 463/929] 10.222.0.16 () {56 vars in 1126 bytes} [Tue Jul 12 17:20:43 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:44.055598 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6463a341-3f5a-4678-bb9b-03de49fd13f2 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:44.059393 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:44.162790 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6463a341-3f5a-4678-bb9b-03de49fd13f2 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:44.184580 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8bb293-afdb-49f4-8eca-679baaad4989 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:44.184960 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8bb293-afdb-49f4-8eca-679baaad4989 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:44.185174 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8bb293-afdb-49f4-8eca-679baaad4989 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:44.186833 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 464/930] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:44.192101 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0f577f-b750-4b7b-9f00-9219dc1659eb None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:44.192399 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0f577f-b750-4b7b-9f00-9219dc1659eb None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:44.192687 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0f577f-b750-4b7b-9f00-9219dc1659eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:44.203747 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13187100-7129-4eda-8fa1-03674623786f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:44.205251 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13187100-7129-4eda-8fa1-03674623786f None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:44.205742 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13187100-7129-4eda-8fa1-03674623786f None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:44.206524 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13187100-7129-4eda-8fa1-03674623786f None admin] PATH_INFO: `/v3/services` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:44.207466 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13187100-7129-4eda-8fa1-03674623786f None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:44.207997 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13187100-7129-4eda-8fa1-03674623786f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:44.209548 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13187100-7129-4eda-8fa1-03674623786f None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:44.210195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-cb0f577f-b750-4b7b-9f00-9219dc1659eb None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:44.211428 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-cb0f577f-b750-4b7b-9f00-9219dc1659eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:44.217118 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 467/931] 10.222.0.16 () {56 vars in 1126 bytes} [Tue Jul 12 17:20:44 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:44.222858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13187100-7129-4eda-8fa1-03674623786f None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:44.299768 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 465/932] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:44 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:20:44.316158 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:44.336501 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb0f577f-b750-4b7b-9f00-9219dc1659eb None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:44.354412 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13187100-7129-4eda-8fa1-03674623786f None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:44.398873 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c61c7f2-08c9-4f7f-8477-9182a1901d45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:44.400267 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c61c7f2-08c9-4f7f-8477-9182a1901d45 None admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:44.401016 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c61c7f2-08c9-4f7f-8477-9182a1901d45 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:44.401451 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c61c7f2-08c9-4f7f-8477-9182a1901d45 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:44.402714 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c61c7f2-08c9-4f7f-8477-9182a1901d45 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:44.404706 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c61c7f2-08c9-4f7f-8477-9182a1901d45 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:44.424774 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:44.457129 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb0f577f-b750-4b7b-9f00-9219dc1659eb None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:44.491113 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e54f9e-1d5b-4496-a449-050ed642513d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:44.491932 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e54f9e-1d5b-4496-a449-050ed642513d None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:44.492222 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e54f9e-1d5b-4496-a449-050ed642513d None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:44.492546 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e54f9e-1d5b-4496-a449-050ed642513d None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:44.503817 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e54f9e-1d5b-4496-a449-050ed642513d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:20:44.512732 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-32e54f9e-1d5b-4496-a449-050ed642513d None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 12 17:20:44.512732 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:44.512732 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:44.512732 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:44.512732 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:44.512732 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:44.512732 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:44.512732 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:44.512732 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:44.512732 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:44.512732 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:44.512732 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:44.512732 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:20:44.512732 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:20:44.512732 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:20:44.512732 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:20:44.513660 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:20:44.513660 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:20:44.513660 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:20:44.513660 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:20:44.513660 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:44.513660 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:44.513660 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:44.513660 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:44.513660 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:44.513660 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:44.513660 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:44.513660 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:44.513660 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:44.513660 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:44.513660 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:44.513660 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:44.519604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:44.519604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:44.519604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:44.519604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:44.519604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:20:44.519604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:20:44.519604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:20:44.519604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:20:44.519604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:20:44.519604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:20:44.519604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 12 17:20:44.519604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:44.519604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 466/933] 10.222.0.16 () {56 vars in 1120 bytes} [Tue Jul 12 17:20:44 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:44.522116 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e54f9e-1d5b-4496-a449-050ed642513d None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:44.546601 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 468/934] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Jul 12 17:20:44 2022] POST /identity/v3/endpoints => generated 347 bytes in 319 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:20:44.648842 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e54f9e-1d5b-4496-a449-050ed642513d None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:44.687759 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:44.698187 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f09d67-7114-44ee-a1df-c3d78f7fcc5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:44.699636 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f09d67-7114-44ee-a1df-c3d78f7fcc5b None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:44.700138 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f09d67-7114-44ee-a1df-c3d78f7fcc5b None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:44.700610 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f09d67-7114-44ee-a1df-c3d78f7fcc5b None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:44.701614 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f09d67-7114-44ee-a1df-c3d78f7fcc5b None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:44.702159 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f09d67-7114-44ee-a1df-c3d78f7fcc5b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:44.703849 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f09d67-7114-44ee-a1df-c3d78f7fcc5b None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:44.713912 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 467/935] 10.222.0.16 () {56 vars in 1115 bytes} [Tue Jul 12 17:20:44 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:20:44.720173 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c61c7f2-08c9-4f7f-8477-9182a1901d45 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:44.759367 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d429d17-6bf4-44e0-8981-bdb9d8773339 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:44.760012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d429d17-6bf4-44e0-8981-bdb9d8773339 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:44.760815 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d429d17-6bf4-44e0-8981-bdb9d8773339 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:44.763463 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 468/936] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:44 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:44.768394 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9299fe4e-9aa6-4baf-9993-dbb5ba530b3a None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:44.768693 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9299fe4e-9aa6-4baf-9993-dbb5ba530b3a None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:44.768982 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9299fe4e-9aa6-4baf-9993-dbb5ba530b3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:44.790994 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-9299fe4e-9aa6-4baf-9993-dbb5ba530b3a None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:44.793445 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-9299fe4e-9aa6-4baf-9993-dbb5ba530b3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:44.854190 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c61c7f2-08c9-4f7f-8477-9182a1901d45 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:44.883887 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:44.885281 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 469/937] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:44 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 12 17:20:44.901444 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730a6318-38de-46ad-b567-fd921ed52d7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:44.902395 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730a6318-38de-46ad-b567-fd921ed52d7b None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:44.902870 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730a6318-38de-46ad-b567-fd921ed52d7b None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:44.903156 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730a6318-38de-46ad-b567-fd921ed52d7b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:44.914821 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-730a6318-38de-46ad-b567-fd921ed52d7b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:44.915238 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9299fe4e-9aa6-4baf-9993-dbb5ba530b3a None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:44.916409 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-730a6318-38de-46ad-b567-fd921ed52d7b None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:44.943452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-730a6318-38de-46ad-b567-fd921ed52d7b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:44.943452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:44.943452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:44.943452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:44.943452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:44.943452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:44.943452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:44.943452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:44.943452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:44.943452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:44.943452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:44.943452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:44.943452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:44.943452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:44.943452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:44.944419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:44.944419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:44.944419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:44.944419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:44.944419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:44.944419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:44.944419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:44.944419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:44.944419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:44.944419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:44.944419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:44.944419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:44.944419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:44.944419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:44.944419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:44.944419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:44.944419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:44.945305 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:44.945305 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:44.945305 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:44.945305 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:44.946508 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 469/938] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:44.949681 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730a6318-38de-46ad-b567-fd921ed52d7b None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:45.047232 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9299fe4e-9aa6-4baf-9993-dbb5ba530b3a None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:45.070058 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:45.085097 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730a6318-38de-46ad-b567-fd921ed52d7b None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:45.094181 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22f1010-ea4e-486b-bae0-43ca6eb0d834 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:45.095030 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22f1010-ea4e-486b-bae0-43ca6eb0d834 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:45.095320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22f1010-ea4e-486b-bae0-43ca6eb0d834 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:45.095603 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22f1010-ea4e-486b-bae0-43ca6eb0d834 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:45.096736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e22f1010-ea4e-486b-bae0-43ca6eb0d834 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:45.098072 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e22f1010-ea4e-486b-bae0-43ca6eb0d834 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:45.112737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-e22f1010-ea4e-486b-bae0-43ca6eb0d834 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 12 17:20:45.112737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:45.112737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 12 17:20:45.112737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 12 17:20:45.112737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 12 17:20:45.112737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 12 17:20:45.112737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 12 17:20:45.112737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:45.112737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 12 17:20:45.112737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:45.112737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 12 17:20:45.112737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:45.112737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:45.112737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:45.112737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:45.112737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:45.112737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:45.113802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:45.113802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:45.113802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:45.113802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:45.113802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:45.113802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:45.113802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 12 17:20:45.113802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 12 17:20:45.113802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 12 17:20:45.113802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 12 17:20:45.113802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:45.113802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:45.113802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:45.113802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:45.113802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:45.113802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:45.114661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:45.114661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:45.114661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:45.114661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:45.114661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:45.114661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:45.114661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:45.114661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:45.114661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:45.114661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:45.114661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 12 17:20:45.114661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:45.114661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 12 17:20:45.114661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:45.119022 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 470/939] 10.222.0.16 () {56 vars in 1126 bytes} [Tue Jul 12 17:20:44 2022] GET /identity/v3/services/object-store => generated 93 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:20:45.119992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22f1010-ea4e-486b-bae0-43ca6eb0d834 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:45.130288 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007a17d4-6110-4128-95e6-52220d85de64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:45.131333 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007a17d4-6110-4128-95e6-52220d85de64 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:45.131914 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007a17d4-6110-4128-95e6-52220d85de64 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:45.132508 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007a17d4-6110-4128-95e6-52220d85de64 None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:45.133523 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007a17d4-6110-4128-95e6-52220d85de64 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:45.134087 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007a17d4-6110-4128-95e6-52220d85de64 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:45.135750 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007a17d4-6110-4128-95e6-52220d85de64 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:45.150174 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 470/940] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:20:45.163753 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007a17d4-6110-4128-95e6-52220d85de64 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:45.210670 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:45.233653 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22f1010-ea4e-486b-bae0-43ca6eb0d834 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:45.272759 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3542d59-1379-4906-ac14-86f8714a9ea0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:45.273433 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3542d59-1379-4906-ac14-86f8714a9ea0 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:45.273722 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3542d59-1379-4906-ac14-86f8714a9ea0 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:45.274108 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3542d59-1379-4906-ac14-86f8714a9ea0 None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:45.274864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3542d59-1379-4906-ac14-86f8714a9ea0 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:45.275392 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3542d59-1379-4906-ac14-86f8714a9ea0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:45.276772 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3542d59-1379-4906-ac14-86f8714a9ea0 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:45.287416 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 471/941] 10.222.0.16 () {56 vars in 1122 bytes} [Tue Jul 12 17:20:45 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:45.288442 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3542d59-1379-4906-ac14-86f8714a9ea0 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:45.296038 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007a17d4-6110-4128-95e6-52220d85de64 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:45.337100 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78439f42-90b5-4cd3-8807-b4f37da21ef9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:45.337958 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78439f42-90b5-4cd3-8807-b4f37da21ef9 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:45.338631 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78439f42-90b5-4cd3-8807-b4f37da21ef9 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:45.338923 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78439f42-90b5-4cd3-8807-b4f37da21ef9 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:45.343214 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:45.348493 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78439f42-90b5-4cd3-8807-b4f37da21ef9 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:45.350102 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78439f42-90b5-4cd3-8807-b4f37da21ef9 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:45.373105 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-78439f42-90b5-4cd3-8807-b4f37da21ef9 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 12 17:20:45.373105 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:45.373105 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:45.373105 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:45.373105 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:45.373105 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:45.373105 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:45.373105 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:45.373105 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:45.373105 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:45.373105 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:45.373105 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:45.373105 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:20:45.373105 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:20:45.373105 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:20:45.373105 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:20:45.374352 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:45.374352 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:45.374352 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:20:45.374352 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:45.374352 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:20:45.374352 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:45.374352 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:45.374352 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:45.374352 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:45.374352 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:45.374352 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:45.374352 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:45.374352 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:45.374352 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:45.374352 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:45.374352 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:45.374352 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:45.374352 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:45.375415 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:45.375415 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:45.375415 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:20:45.375415 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:20:45.375415 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:20:45.375415 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:20:45.375415 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:20:45.375415 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:20:45.375415 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 12 17:20:45.375415 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:45.378129 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 471/942] 10.222.0.16 () {56 vars in 1117 bytes} [Tue Jul 12 17:20:45 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:45.384620 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78439f42-90b5-4cd3-8807-b4f37da21ef9 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:45.408215 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3542d59-1379-4906-ac14-86f8714a9ea0 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:45.446664 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e50d44-779d-4f4c-90ba-1926dc07f75c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:45.447420 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e50d44-779d-4f4c-90ba-1926dc07f75c None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:45.447787 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e50d44-779d-4f4c-90ba-1926dc07f75c None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:45.448145 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e50d44-779d-4f4c-90ba-1926dc07f75c None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:45.448933 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64e50d44-779d-4f4c-90ba-1926dc07f75c None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:45.449430 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64e50d44-779d-4f4c-90ba-1926dc07f75c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:45.451039 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64e50d44-779d-4f4c-90ba-1926dc07f75c None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:45.461360 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 472/943] 10.222.0.16 () {56 vars in 1122 bytes} [Tue Jul 12 17:20:45 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:45.463046 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e50d44-779d-4f4c-90ba-1926dc07f75c None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:45.516575 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:45.526603 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78439f42-90b5-4cd3-8807-b4f37da21ef9 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:45.565814 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc9c1a5-bbe8-485e-852a-1a08ee5a07bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:45.565814 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc9c1a5-bbe8-485e-852a-1a08ee5a07bb None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:45.565814 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc9c1a5-bbe8-485e-852a-1a08ee5a07bb None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:45.565814 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dc9c1a5-bbe8-485e-852a-1a08ee5a07bb None admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:45.566281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc9c1a5-bbe8-485e-852a-1a08ee5a07bb None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:45.566281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc9c1a5-bbe8-485e-852a-1a08ee5a07bb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:45.567295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dc9c1a5-bbe8-485e-852a-1a08ee5a07bb None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:45.590186 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 472/944] 10.222.0.16 () {56 vars in 1149 bytes} [Tue Jul 12 17:20:45 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:45.591583 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc9c1a5-bbe8-485e-852a-1a08ee5a07bb None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:45.594008 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e50d44-779d-4f4c-90ba-1926dc07f75c None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:45.606911 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:45.634472 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4fbd29-de25-4f8d-8aef-099bb7ee83ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:45.635405 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4fbd29-de25-4f8d-8aef-099bb7ee83ed None admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:45.635908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4fbd29-de25-4f8d-8aef-099bb7ee83ed None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:45.636377 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4fbd29-de25-4f8d-8aef-099bb7ee83ed None admin] PATH_INFO: `/v3/endpoints` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:45.637877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc4fbd29-de25-4f8d-8aef-099bb7ee83ed None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:45.639644 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc4fbd29-de25-4f8d-8aef-099bb7ee83ed None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:45.698830 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dc9c1a5-bbe8-485e-852a-1a08ee5a07bb None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:45.717445 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 473/945] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Jul 12 17:20:45 2022] POST /identity/v3/endpoints => generated 349 bytes in 255 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:20:45.719054 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3f338c-aaad-4ec4-90bc-34c84d64f0ce None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:45.719348 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3f338c-aaad-4ec4-90bc-34c84d64f0ce None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:45.719629 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a3f338c-aaad-4ec4-90bc-34c84d64f0ce None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:45.721188 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 474/946] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:45.727018 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3397247b-7f4e-47b3-980b-ceb798751b06 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:45.727328 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3397247b-7f4e-47b3-980b-ceb798751b06 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:45.727592 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3397247b-7f4e-47b3-980b-ceb798751b06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:45.746830 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-3397247b-7f4e-47b3-980b-ceb798751b06 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:45.748205 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-3397247b-7f4e-47b3-980b-ceb798751b06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:45.757931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bc420a-eb2b-48bc-a350-57f4e0aff5c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:45.758805 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bc420a-eb2b-48bc-a350-57f4e0aff5c1 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:45.759193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bc420a-eb2b-48bc-a350-57f4e0aff5c1 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:45.762400 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bc420a-eb2b-48bc-a350-57f4e0aff5c1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:45.773167 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92bc420a-eb2b-48bc-a350-57f4e0aff5c1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:45.774690 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92bc420a-eb2b-48bc-a350-57f4e0aff5c1 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:45.786280 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:45.796689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-92bc420a-eb2b-48bc-a350-57f4e0aff5c1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:45.796689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:45.796689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:45.796689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:45.796689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:45.796689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:45.796689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:45.796689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:45.796689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:45.796689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:45.796689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:45.796689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:45.796689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:45.796689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:45.796689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:45.804210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:45.804210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:45.804210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:45.804210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:45.804210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:45.804210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:45.804210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:45.804210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:45.804210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:45.804210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:45.804210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:45.804210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:45.804210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:45.804210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:45.804210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:45.804210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:45.804210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:45.805109 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:45.805109 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:45.805109 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:45.805109 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:45.805109 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 473/947] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:45.805109 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bc420a-eb2b-48bc-a350-57f4e0aff5c1 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:45.854612 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 475/948] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:45 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 12 17:20:45.879373 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3397247b-7f4e-47b3-980b-ceb798751b06 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:45.928723 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bc420a-eb2b-48bc-a350-57f4e0aff5c1 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:45.968446 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ecd688-c490-49c8-9eb1-f4d9780b677b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:45.969036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ecd688-c490-49c8-9eb1-f4d9780b677b None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:45.969379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ecd688-c490-49c8-9eb1-f4d9780b677b None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:45.969747 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ecd688-c490-49c8-9eb1-f4d9780b677b None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:45.971127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ecd688-c490-49c8-9eb1-f4d9780b677b None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:45.971127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ecd688-c490-49c8-9eb1-f4d9780b677b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:45.972194 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42ecd688-c490-49c8-9eb1-f4d9780b677b None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:45.985193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 474/949] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:45.991549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ecd688-c490-49c8-9eb1-f4d9780b677b None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:45.999527 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:46.005895 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3397247b-7f4e-47b3-980b-ceb798751b06 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:46.048471 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe53f689-818e-4462-83db-4cab7d2527cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:46.049720 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe53f689-818e-4462-83db-4cab7d2527cf None admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:46.050191 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe53f689-818e-4462-83db-4cab7d2527cf None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:46.050968 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe53f689-818e-4462-83db-4cab7d2527cf None admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:46.052347 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe53f689-818e-4462-83db-4cab7d2527cf None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:46.053965 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe53f689-818e-4462-83db-4cab7d2527cf None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:46.056162 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: WARNING keystone.server.flask.common [None req-fe53f689-818e-4462-83db-4cab7d2527cf None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Jul 12 17:20:46.063724 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:46.107665 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ecd688-c490-49c8-9eb1-f4d9780b677b None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:46.141355 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b40bd7c-c0a7-48c9-b97b-04597884b5d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:46.142147 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b40bd7c-c0a7-48c9-b97b-04597884b5d8 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:46.142453 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b40bd7c-c0a7-48c9-b97b-04597884b5d8 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:46.142757 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b40bd7c-c0a7-48c9-b97b-04597884b5d8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:46.149516 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b40bd7c-c0a7-48c9-b97b-04597884b5d8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:46.150992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b40bd7c-c0a7-48c9-b97b-04597884b5d8 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:46.165430 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-6b40bd7c-c0a7-48c9-b97b-04597884b5d8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:46.165430 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:46.165430 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:46.165430 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:46.165430 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:46.165430 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:46.165430 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:46.165430 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:46.165430 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:46.165430 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:46.165430 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:46.165430 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:46.165430 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:20:46.165430 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:20:46.165430 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:20:46.167035 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:20:46.167035 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:46.167035 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:46.167035 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:46.167035 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:46.167035 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:46.167035 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:46.167035 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:46.167035 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:46.167035 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:46.167035 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:46.167035 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:46.167035 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:46.167035 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:46.167035 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:46.167035 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:46.167035 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:46.168822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:20:46.168822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:20:46.168822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:20:46.168822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:20:46.168822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:20:46.168822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:20:46.168822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:46.168822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:46.171226 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.sql.core [None req-fe53f689-818e-4462-83db-4cab7d2527cf None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Jul 12 17:20:46.171226 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Jul 12 17:20:46.171226 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [parameters: {'id': 'd3e8ae27b60d43539fe55a116056cc60', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Jul 12 17:20:46.171226 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=80452) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Jul 12 17:20:46.172231 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 475/950] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:20:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:20:46.173491 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b40bd7c-c0a7-48c9-b97b-04597884b5d8 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:46.182901 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-fe53f689-818e-4462-83db-4cab7d2527cf None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id d3e8ae27b60d43539fe55a116056cc60.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id d3e8ae27b60d43539fe55a116056cc60. Jul 12 17:20:46.182901 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:46.182901 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 12 17:20:46.182901 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 12 17:20:46.182901 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 12 17:20:46.182901 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 12 17:20:46.182901 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 12 17:20:46.182901 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application result = self._query(query) Jul 12 17:20:46.182901 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 12 17:20:46.182901 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application conn.query(q) Jul 12 17:20:46.182901 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 12 17:20:46.182901 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 12 17:20:46.182901 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 12 17:20:46.184632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application result.read() Jul 12 17:20:46.184632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 12 17:20:46.184632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 12 17:20:46.184632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 12 17:20:46.184632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 12 17:20:46.184632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 12 17:20:46.184632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 12 17:20:46.184632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 12 17:20:46.184632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 12 17:20:46.184632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Jul 12 17:20:46.184632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:46.184632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Jul 12 17:20:46.184632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:46.184632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:46.184632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Jul 12 17:20:46.184632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return method(*args, **kwargs) Jul 12 17:20:46.184632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Jul 12 17:20:46.184632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return project_ref.to_dict() Jul 12 17:20:46.185736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 12 17:20:46.185736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application next(self.gen) Jul 12 17:20:46.185736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Jul 12 17:20:46.185736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application yield resource Jul 12 17:20:46.185736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 12 17:20:46.185736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application next(self.gen) Jul 12 17:20:46.185736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Jul 12 17:20:46.185736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self.session.rollback() Jul 12 17:20:46.185736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jul 12 17:20:46.185736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self.force_reraise() Jul 12 17:20:46.185736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jul 12 17:20:46.185736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise self.value Jul 12 17:20:46.185736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Jul 12 17:20:46.185736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Jul 12 17:20:46.185736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Jul 12 17:20:46.185736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application session.commit() Jul 12 17:20:46.185736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Jul 12 17:20:46.186727 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Jul 12 17:20:46.186727 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Jul 12 17:20:46.186727 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self._prepare_impl() Jul 12 17:20:46.186727 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Jul 12 17:20:46.186727 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self.session.flush() Jul 12 17:20:46.186727 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Jul 12 17:20:46.186727 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self._flush(objects) Jul 12 17:20:46.186727 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Jul 12 17:20:46.186727 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Jul 12 17:20:46.186727 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Jul 12 17:20:46.186727 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application compat.raise_( Jul 12 17:20:46.186727 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 12 17:20:46.186727 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception Jul 12 17:20:46.186727 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Jul 12 17:20:46.186727 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application flush_context.execute() Jul 12 17:20:46.186727 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Jul 12 17:20:46.186727 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rec.execute(self) Jul 12 17:20:46.187626 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Jul 12 17:20:46.187626 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Jul 12 17:20:46.187626 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Jul 12 17:20:46.187626 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application _emit_insert_statements( Jul 12 17:20:46.187626 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Jul 12 17:20:46.187626 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application c = connection._execute_20( Jul 12 17:20:46.187626 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 Jul 12 17:20:46.187626 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Jul 12 17:20:46.187626 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 332, in _execute_on_connection Jul 12 17:20:46.187626 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Jul 12 17:20:46.187626 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement Jul 12 17:20:46.187626 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ret = self._execute_context( Jul 12 17:20:46.187626 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context Jul 12 17:20:46.187626 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Jul 12 17:20:46.187626 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2041, in _handle_dbapi_exception Jul 12 17:20:46.188427 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Jul 12 17:20:46.188427 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 12 17:20:46.188427 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception Jul 12 17:20:46.188427 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 12 17:20:46.188427 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 12 17:20:46.188427 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 12 17:20:46.188427 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 12 17:20:46.188427 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 12 17:20:46.188427 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application result = self._query(query) Jul 12 17:20:46.188427 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 12 17:20:46.188427 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application conn.query(q) Jul 12 17:20:46.188427 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 12 17:20:46.188427 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 12 17:20:46.188427 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 12 17:20:46.188427 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application result.read() Jul 12 17:20:46.188427 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 12 17:20:46.188427 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 12 17:20:46.189283 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 12 17:20:46.189283 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 12 17:20:46.189283 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 12 17:20:46.189283 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 12 17:20:46.189283 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 12 17:20:46.189283 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 12 17:20:46.189283 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Jul 12 17:20:46.189283 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Jul 12 17:20:46.189283 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [parameters: {'id': 'd3e8ae27b60d43539fe55a116056cc60', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Jul 12 17:20:46.189283 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Jul 12 17:20:46.189283 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:46.189283 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 12 17:20:46.189283 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:46.189283 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:46.189283 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Jul 12 17:20:46.191856 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Jul 12 17:20:46.191856 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Jul 12 17:20:46.191856 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Jul 12 17:20:46.191856 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Jul 12 17:20:46.191856 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:46.191856 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 12 17:20:46.191856 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:46.191856 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:46.191856 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:46.191856 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:46.191856 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:46.191856 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:46.191856 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:46.191856 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:46.191856 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:46.191856 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:46.191856 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:46.194021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:46.194021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Jul 12 17:20:46.194021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Jul 12 17:20:46.194021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:46.194021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:46.194021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Jul 12 17:20:46.194021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.Conflict( Jul 12 17:20:46.194021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id d3e8ae27b60d43539fe55a116056cc60. Jul 12 17:20:46.194021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:46.195670 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 476/951] 10.222.0.16 () {60 vars in 1140 bytes} [Tue Jul 12 17:20:45 2022] POST /identity/v3/projects => generated 262 bytes in 317 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:20:46.203701 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe53f689-818e-4462-83db-4cab7d2527cf None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:46.295478 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b40bd7c-c0a7-48c9-b97b-04597884b5d8 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:46.321871 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:46.334225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe53f689-818e-4462-83db-4cab7d2527cf None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:46.342268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc7df77-c8b6-49e5-a32e-7405243fa4ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:46.343603 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc7df77-c8b6-49e5-a32e-7405243fa4ef None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:46.343958 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc7df77-c8b6-49e5-a32e-7405243fa4ef None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:46.344300 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc7df77-c8b6-49e5-a32e-7405243fa4ef None admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:46.345157 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc7df77-c8b6-49e5-a32e-7405243fa4ef None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:46.345536 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc7df77-c8b6-49e5-a32e-7405243fa4ef None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:46.348106 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc7df77-c8b6-49e5-a32e-7405243fa4ef None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:46.366458 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 476/952] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Jul 12 17:20:46 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:20:46.372858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc7df77-c8b6-49e5-a32e-7405243fa4ef None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:46.376584 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089c3a43-e83c-4108-be92-54c894803e99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:46.377497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089c3a43-e83c-4108-be92-54c894803e99 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:46.377884 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089c3a43-e83c-4108-be92-54c894803e99 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:46.378264 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089c3a43-e83c-4108-be92-54c894803e99 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:46.387349 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-089c3a43-e83c-4108-be92-54c894803e99 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:46.389721 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-089c3a43-e83c-4108-be92-54c894803e99 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:46.410924 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-089c3a43-e83c-4108-be92-54c894803e99 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:46.410924 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:46.410924 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:46.410924 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:46.410924 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:46.410924 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:46.410924 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:46.410924 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:46.410924 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:46.410924 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:46.410924 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:46.410924 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:46.410924 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:20:46.410924 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:20:46.410924 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:20:46.412527 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:20:46.412527 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:46.412527 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:46.412527 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:46.412527 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:46.412527 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:46.412527 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:46.412527 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:46.412527 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:46.412527 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:46.412527 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:46.412527 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:46.412527 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:46.412527 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:46.412527 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:46.412527 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:46.412527 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:46.413679 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:20:46.413679 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:20:46.413679 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:20:46.413679 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:20:46.413679 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:20:46.413679 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:20:46.413679 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:46.413679 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:46.420272 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 477/953] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:20:46 2022] GET /identity/v3/projects/service => generated 88 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:46.421628 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089c3a43-e83c-4108-be92-54c894803e99 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:46.442218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:46.498480 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc7df77-c8b6-49e5-a32e-7405243fa4ef None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:46.535236 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2385390-38c2-44f9-a500-7032d822fcdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:46.536471 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2385390-38c2-44f9-a500-7032d822fcdb None admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:46.538205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2385390-38c2-44f9-a500-7032d822fcdb None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:46.538829 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2385390-38c2-44f9-a500-7032d822fcdb None admin] PATH_INFO: `/v3/projects/fd4ed2fe037c4fb7ba49c724eb9414c2/users/3ac0d5d8de4943a3888abee3b24b8a6f/roles/feb6a96f7b6c4f0a8d64be226f85ee6c` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:46.545037 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2385390-38c2-44f9-a500-7032d822fcdb None admin] RBAC: Authorizing `identity:create_grant(project_id=fd4ed2fe037c4fb7ba49c724eb9414c2, user_id=3ac0d5d8de4943a3888abee3b24b8a6f, role_id=feb6a96f7b6c4f0a8d64be226f85ee6c)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:46.550523 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2385390-38c2-44f9-a500-7032d822fcdb None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:46.556915 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089c3a43-e83c-4108-be92-54c894803e99 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:46.600705 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43801b9c-4cfe-4ae3-bfe8-2cc44cc3e3e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:46.601513 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43801b9c-4cfe-4ae3-bfe8-2cc44cc3e3e4 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:46.602004 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43801b9c-4cfe-4ae3-bfe8-2cc44cc3e3e4 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:46.602389 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43801b9c-4cfe-4ae3-bfe8-2cc44cc3e3e4 None admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:46.603287 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43801b9c-4cfe-4ae3-bfe8-2cc44cc3e3e4 None admin] RBAC: Adding query filter params (name=service) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:46.603706 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43801b9c-4cfe-4ae3-bfe8-2cc44cc3e3e4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:46.605921 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43801b9c-4cfe-4ae3-bfe8-2cc44cc3e3e4 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:46.627487 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 478/954] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Jul 12 17:20:46 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:46.645262 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 477/955] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Jul 12 17:20:46 2022] PUT /identity/v3/projects/fd4ed2fe037c4fb7ba49c724eb9414c2/users/3ac0d5d8de4943a3888abee3b24b8a6f/roles/feb6a96f7b6c4f0a8d64be226f85ee6c => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:20:46.808895 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f64b0f81-69ec-4557-a7e4-c6d59486de85 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:46.809279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f64b0f81-69ec-4557-a7e4-c6d59486de85 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:46.809650 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f64b0f81-69ec-4557-a7e4-c6d59486de85 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:46.811502 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 479/956] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:46.816931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3ec0e4-b3f1-47b8-a5ba-5d847594f88f None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:46.817240 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3ec0e4-b3f1-47b8-a5ba-5d847594f88f None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:46.817523 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3ec0e4-b3f1-47b8-a5ba-5d847594f88f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:46.838218 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-7c3ec0e4-b3f1-47b8-a5ba-5d847594f88f None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:46.839462 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-7c3ec0e4-b3f1-47b8-a5ba-5d847594f88f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:46.878764 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:46.913989 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 478/957] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:46 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:20:46.937799 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f64b0f81-69ec-4557-a7e4-c6d59486de85 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:47.020671 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:47.053522 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f64b0f81-69ec-4557-a7e4-c6d59486de85 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:47.089492 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2c7863-6b89-406f-89ef-d1aa4918b06c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:47.090220 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2c7863-6b89-406f-89ef-d1aa4918b06c None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:47.090544 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2c7863-6b89-406f-89ef-d1aa4918b06c None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:47.090846 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd2c7863-6b89-406f-89ef-d1aa4918b06c None admin] PATH_INFO: `/v3/services/compute` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:47.091477 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd2c7863-6b89-406f-89ef-d1aa4918b06c None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:47.092995 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd2c7863-6b89-406f-89ef-d1aa4918b06c None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:47.105930 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-bd2c7863-6b89-406f-89ef-d1aa4918b06c None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Jul 12 17:20:47.105930 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:47.105930 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 12 17:20:47.105930 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 12 17:20:47.105930 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 12 17:20:47.105930 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 12 17:20:47.105930 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 12 17:20:47.105930 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:47.105930 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 12 17:20:47.105930 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:47.105930 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 12 17:20:47.105930 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:47.105930 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:47.105930 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:47.105930 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:47.105930 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:47.105930 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:47.107183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:47.107183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:47.107183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:47.107183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:47.107183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:47.107183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:47.107183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 12 17:20:47.107183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 12 17:20:47.107183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 12 17:20:47.107183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 12 17:20:47.107183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:47.107183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:47.107183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:47.107183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:47.107183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:47.107183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:47.108012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:47.108012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:47.108012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:47.108012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:47.108012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:47.108012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:47.108012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:47.108012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:47.108012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:47.108012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:47.108012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 12 17:20:47.108012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:47.108012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 12 17:20:47.108012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:47.108012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 480/958] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:20:46 2022] GET /identity/v3/services/compute => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:47.113585 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3ec0e4-b3f1-47b8-a5ba-5d847594f88f None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:47.215072 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3ec0e4-b3f1-47b8-a5ba-5d847594f88f None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:47.250311 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b18173-83d7-41bf-be30-2990036440c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:47.251021 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b18173-83d7-41bf-be30-2990036440c2 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:47.251312 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b18173-83d7-41bf-be30-2990036440c2 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:47.251596 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b18173-83d7-41bf-be30-2990036440c2 None admin] PATH_INFO: `/v3/services` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:47.252316 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b18173-83d7-41bf-be30-2990036440c2 None admin] RBAC: Adding query filter params (name=compute) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:47.252631 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b18173-83d7-41bf-be30-2990036440c2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:47.254004 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b18173-83d7-41bf-be30-2990036440c2 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:47.259366 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 479/959] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Jul 12 17:20:47 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:47.264971 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2c7863-6b89-406f-89ef-d1aa4918b06c None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:47.372506 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd2c7863-6b89-406f-89ef-d1aa4918b06c None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:47.407911 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53f3760-bfd7-4670-b479-d3972983202a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:47.408572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53f3760-bfd7-4670-b479-d3972983202a None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:47.408859 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53f3760-bfd7-4670-b479-d3972983202a None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:47.409140 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53f3760-bfd7-4670-b479-d3972983202a None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:47.409860 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53f3760-bfd7-4670-b479-d3972983202a None admin] RBAC: Adding query filter params (type=compute) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:47.410174 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53f3760-bfd7-4670-b479-d3972983202a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:47.411581 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53f3760-bfd7-4670-b479-d3972983202a None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:47.413890 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:47.416850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 481/960] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Jul 12 17:20:47 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:47.920137 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7c9adf-ad74-4c9e-abf3-32c47bf0a305 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:47.920565 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7c9adf-ad74-4c9e-abf3-32c47bf0a305 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:47.920744 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e7c9adf-ad74-4c9e-abf3-32c47bf0a305 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:47.922483 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 480/961] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:47.928630 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0b4a0b-9d5e-46fc-93cc-d138383a86cb None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:47.928951 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0b4a0b-9d5e-46fc-93cc-d138383a86cb None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:47.929206 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0b4a0b-9d5e-46fc-93cc-d138383a86cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:47.948109 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-cd0b4a0b-9d5e-46fc-93cc-d138383a86cb None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:47.950198 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-cd0b4a0b-9d5e-46fc-93cc-d138383a86cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:48.021918 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 482/962] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:47 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:20:48.044400 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7c9adf-ad74-4c9e-abf3-32c47bf0a305 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:48.081806 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:48.151061 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e7c9adf-ad74-4c9e-abf3-32c47bf0a305 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:48.191111 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4f4ec6-8ad7-4c43-9163-9e3e93badd5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:48.191833 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4f4ec6-8ad7-4c43-9163-9e3e93badd5e None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:48.192118 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4f4ec6-8ad7-4c43-9163-9e3e93badd5e None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:48.192397 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4f4ec6-8ad7-4c43-9163-9e3e93badd5e None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:48.193053 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f4f4ec6-8ad7-4c43-9163-9e3e93badd5e None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:48.194410 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f4f4ec6-8ad7-4c43-9163-9e3e93badd5e None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:48.207183 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-8f4f4ec6-8ad7-4c43-9163-9e3e93badd5e None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 12 17:20:48.207183 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:48.207183 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 12 17:20:48.207183 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 12 17:20:48.207183 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 12 17:20:48.207183 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 12 17:20:48.207183 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 12 17:20:48.207183 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:48.207183 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 12 17:20:48.207183 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:48.207183 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 12 17:20:48.207183 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:48.207183 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:48.207183 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:48.207183 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:48.207183 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:48.207183 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:48.208288 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:48.208288 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:48.208288 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:48.208288 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:48.208288 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:48.208288 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:48.208288 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 12 17:20:48.208288 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 12 17:20:48.208288 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 12 17:20:48.208288 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 12 17:20:48.208288 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:48.208288 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:48.208288 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:48.208288 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:48.208288 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:48.208288 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:48.209143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:48.209143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:48.209143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:48.209143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:48.209143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:48.209143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:48.209143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:48.209143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:48.209143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:48.209143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:48.209143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 12 17:20:48.209143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:48.209143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 12 17:20:48.209143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:48.209143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 481/963] 10.222.0.16 () {56 vars in 1126 bytes} [Tue Jul 12 17:20:48 2022] GET /identity/v3/services/object-store => generated 93 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:48.215157 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0b4a0b-9d5e-46fc-93cc-d138383a86cb None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:48.320753 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0b4a0b-9d5e-46fc-93cc-d138383a86cb None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:48.351756 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:48.358156 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a211de-958b-4f56-a5cb-16930aadb9b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:48.358868 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a211de-958b-4f56-a5cb-16930aadb9b5 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:48.359157 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a211de-958b-4f56-a5cb-16930aadb9b5 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:48.359438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a211de-958b-4f56-a5cb-16930aadb9b5 None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:48.360160 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a211de-958b-4f56-a5cb-16930aadb9b5 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:48.360474 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a211de-958b-4f56-a5cb-16930aadb9b5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:48.361907 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a211de-958b-4f56-a5cb-16930aadb9b5 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:48.367148 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 483/964] 10.222.0.16 () {56 vars in 1122 bytes} [Tue Jul 12 17:20:48 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:48.373736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4f4ec6-8ad7-4c43-9163-9e3e93badd5e None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:48.475026 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:48.479136 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4f4ec6-8ad7-4c43-9163-9e3e93badd5e None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:48.516055 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943ffec7-a752-4689-9729-a8dea2bdb128 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:48.516736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943ffec7-a752-4689-9729-a8dea2bdb128 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:48.517024 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943ffec7-a752-4689-9729-a8dea2bdb128 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:48.517307 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943ffec7-a752-4689-9729-a8dea2bdb128 None admin] PATH_INFO: `/v3/services` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:48.518024 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-943ffec7-a752-4689-9729-a8dea2bdb128 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:48.518335 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-943ffec7-a752-4689-9729-a8dea2bdb128 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:48.519742 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-943ffec7-a752-4689-9729-a8dea2bdb128 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:48.525445 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 482/965] 10.222.0.16 () {56 vars in 1122 bytes} [Tue Jul 12 17:20:48 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:48.531328 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a211de-958b-4f56-a5cb-16930aadb9b5 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:48.653523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a211de-958b-4f56-a5cb-16930aadb9b5 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:48.689133 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce40f784-7bbf-4b51-bca3-edbd8ee9c442 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:48.689797 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce40f784-7bbf-4b51-bca3-edbd8ee9c442 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:48.690086 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce40f784-7bbf-4b51-bca3-edbd8ee9c442 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:48.690365 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce40f784-7bbf-4b51-bca3-edbd8ee9c442 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:48.691202 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce40f784-7bbf-4b51-bca3-edbd8ee9c442 None admin] RBAC: Adding query filter params (interface=admin, service_id=2b75d6e9520b406e8298d5868e24fc50, region_id=RegionOne) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:48.691520 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce40f784-7bbf-4b51-bca3-edbd8ee9c442 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:48.692881 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce40f784-7bbf-4b51-bca3-edbd8ee9c442 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:48.699600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 484/966] 10.222.0.16 () {56 vars in 1249 bytes} [Tue Jul 12 17:20:48 2022] GET /identity/v3/endpoints?service_id=2b75d6e9520b406e8298d5868e24fc50&interface=admin®ion_id=RegionOne => generated 193 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:48.705008 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943ffec7-a752-4689-9729-a8dea2bdb128 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:48.813727 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943ffec7-a752-4689-9729-a8dea2bdb128 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:48.850133 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cba1ef2-41d2-4dcb-b8c7-e9757e5c1d57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:48.851088 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cba1ef2-41d2-4dcb-b8c7-e9757e5c1d57 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:48.851420 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cba1ef2-41d2-4dcb-b8c7-e9757e5c1d57 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:48.851743 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cba1ef2-41d2-4dcb-b8c7-e9757e5c1d57 None admin] PATH_INFO: `/v3/services` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:48.852438 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cba1ef2-41d2-4dcb-b8c7-e9757e5c1d57 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:48.853805 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cba1ef2-41d2-4dcb-b8c7-e9757e5c1d57 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:48.860699 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 483/967] 10.222.0.16 () {56 vars in 1087 bytes} [Tue Jul 12 17:20:48 2022] GET /identity/v3/services => generated 1497 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:20:48.884685 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cefd4e-c7bc-4603-b155-2a2df5932efc None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:48.885014 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cefd4e-c7bc-4603-b155-2a2df5932efc None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:48.885257 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cefd4e-c7bc-4603-b155-2a2df5932efc None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:48.887665 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 485/968] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:48.895362 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95efbf41-1c61-4b9a-a477-4a02fa0d4dc4 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:48.895701 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95efbf41-1c61-4b9a-a477-4a02fa0d4dc4 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:48.895945 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95efbf41-1c61-4b9a-a477-4a02fa0d4dc4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:48.914813 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-95efbf41-1c61-4b9a-a477-4a02fa0d4dc4 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:48.916061 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-95efbf41-1c61-4b9a-a477-4a02fa0d4dc4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:48.940848 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:48.962201 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38894e42-d2ce-48f6-a1da-7a1c7ef86061 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:48.962614 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38894e42-d2ce-48f6-a1da-7a1c7ef86061 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:48.962966 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38894e42-d2ce-48f6-a1da-7a1c7ef86061 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:48.964805 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 486/969] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:48.971158 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d29949-1c6c-403b-aa17-0268497e8ff8 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:48.971515 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d29949-1c6c-403b-aa17-0268497e8ff8 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:48.971892 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d29949-1c6c-403b-aa17-0268497e8ff8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:48.991138 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-18d29949-1c6c-403b-aa17-0268497e8ff8 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:48.992660 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-18d29949-1c6c-403b-aa17-0268497e8ff8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:49.000185 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 484/970] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:48 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:20:49.031025 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95efbf41-1c61-4b9a-a477-4a02fa0d4dc4 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:49.054668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:49.071699 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 487/971] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:48 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 12 17:20:49.103458 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d29949-1c6c-403b-aa17-0268497e8ff8 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:49.152613 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95efbf41-1c61-4b9a-a477-4a02fa0d4dc4 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:49.189555 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c219316f-319d-4fd9-b86f-c41c5eb2d928 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:49.190829 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c219316f-319d-4fd9-b86f-c41c5eb2d928 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:49.191374 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c219316f-319d-4fd9-b86f-c41c5eb2d928 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:49.191862 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c219316f-319d-4fd9-b86f-c41c5eb2d928 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:49.204340 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c219316f-319d-4fd9-b86f-c41c5eb2d928 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:20:49.213655 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-c219316f-319d-4fd9-b86f-c41c5eb2d928 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 12 17:20:49.213655 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:49.213655 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:49.213655 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:49.213655 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:49.213655 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:49.213655 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:49.213655 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:49.213655 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:49.213655 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:49.213655 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:49.213655 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:49.213655 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:20:49.213655 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:20:49.213655 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:20:49.213655 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:20:49.215215 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:20:49.215215 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:20:49.215215 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:20:49.215215 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:20:49.215215 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:49.215215 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:49.215215 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:49.215215 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:49.215215 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:49.215215 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:49.215215 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:49.215215 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:49.215215 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:49.215215 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:49.215215 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:49.215215 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:49.216083 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:49.216083 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:49.216083 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:49.216083 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:49.216083 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:20:49.216083 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:20:49.216083 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:20:49.216083 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:20:49.216083 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:20:49.216083 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:20:49.216083 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 12 17:20:49.216083 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:49.219007 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 485/972] 10.222.0.16 () {56 vars in 1120 bytes} [Tue Jul 12 17:20:49 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:20:49.226607 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c219316f-319d-4fd9-b86f-c41c5eb2d928 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:49.233747 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d29949-1c6c-403b-aa17-0268497e8ff8 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:49.280413 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64af076a-3baf-4d27-bf3b-e4a7c6d8c70d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:49.281103 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64af076a-3baf-4d27-bf3b-e4a7c6d8c70d None admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:49.281389 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64af076a-3baf-4d27-bf3b-e4a7c6d8c70d None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:49.281669 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64af076a-3baf-4d27-bf3b-e4a7c6d8c70d None admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:49.282717 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64af076a-3baf-4d27-bf3b-e4a7c6d8c70d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:49.284210 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64af076a-3baf-4d27-bf3b-e4a7c6d8c70d None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:49.293602 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:49.335062 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c219316f-319d-4fd9-b86f-c41c5eb2d928 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:49.361873 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.sql.core [None req-64af076a-3baf-4d27-bf3b-e4a7c6d8c70d None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Jul 12 17:20:49.361873 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Jul 12 17:20:49.361873 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [parameters: {'user_id': '1a81453c45404c5da9c085e0e7e2e949', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Jul 12 17:20:49.361873 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=80452) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Jul 12 17:20:49.369235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-64af076a-3baf-4d27-bf3b-e4a7c6d8c70d None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Jul 12 17:20:49.369235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:49.369235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 12 17:20:49.369235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 12 17:20:49.369235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 12 17:20:49.369235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 12 17:20:49.369235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 12 17:20:49.369235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application result = self._query(query) Jul 12 17:20:49.369235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 12 17:20:49.369235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application conn.query(q) Jul 12 17:20:49.369235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 12 17:20:49.369235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 12 17:20:49.369235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 12 17:20:49.369235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application result.read() Jul 12 17:20:49.369235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 12 17:20:49.370161 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 12 17:20:49.370161 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 12 17:20:49.370161 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 12 17:20:49.370161 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 12 17:20:49.370161 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 12 17:20:49.370161 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 12 17:20:49.370161 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 12 17:20:49.370161 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Jul 12 17:20:49.370161 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:49.370161 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Jul 12 17:20:49.370161 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:49.370161 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:49.370161 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Jul 12 17:20:49.370161 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return method(*args, **kwargs) Jul 12 17:20:49.370161 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Jul 12 17:20:49.370161 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Jul 12 17:20:49.370161 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 12 17:20:49.370161 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application next(self.gen) Jul 12 17:20:49.371238 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Jul 12 17:20:49.371238 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application yield resource Jul 12 17:20:49.371238 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 12 17:20:49.371238 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application next(self.gen) Jul 12 17:20:49.371238 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Jul 12 17:20:49.371238 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self.session.rollback() Jul 12 17:20:49.371238 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jul 12 17:20:49.371238 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self.force_reraise() Jul 12 17:20:49.371238 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jul 12 17:20:49.371238 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise self.value Jul 12 17:20:49.371238 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Jul 12 17:20:49.371238 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Jul 12 17:20:49.371238 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Jul 12 17:20:49.371238 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application session.commit() Jul 12 17:20:49.371238 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Jul 12 17:20:49.371238 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Jul 12 17:20:49.371238 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Jul 12 17:20:49.372202 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self._prepare_impl() Jul 12 17:20:49.372202 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Jul 12 17:20:49.372202 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self.session.flush() Jul 12 17:20:49.372202 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Jul 12 17:20:49.372202 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self._flush(objects) Jul 12 17:20:49.372202 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Jul 12 17:20:49.372202 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Jul 12 17:20:49.372202 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Jul 12 17:20:49.372202 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application compat.raise_( Jul 12 17:20:49.372202 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 12 17:20:49.372202 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception Jul 12 17:20:49.372202 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Jul 12 17:20:49.372202 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application flush_context.execute() Jul 12 17:20:49.372202 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Jul 12 17:20:49.372202 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rec.execute(self) Jul 12 17:20:49.372202 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Jul 12 17:20:49.372202 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Jul 12 17:20:49.373119 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Jul 12 17:20:49.373119 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application _emit_insert_statements( Jul 12 17:20:49.373119 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Jul 12 17:20:49.373119 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application result = connection._execute_20( Jul 12 17:20:49.373119 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 Jul 12 17:20:49.373119 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Jul 12 17:20:49.373119 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 332, in _execute_on_connection Jul 12 17:20:49.373119 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Jul 12 17:20:49.373119 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement Jul 12 17:20:49.373119 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ret = self._execute_context( Jul 12 17:20:49.373119 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context Jul 12 17:20:49.373119 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Jul 12 17:20:49.373119 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2041, in _handle_dbapi_exception Jul 12 17:20:49.373119 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Jul 12 17:20:49.373119 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 12 17:20:49.373119 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception Jul 12 17:20:49.373992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 12 17:20:49.373992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 12 17:20:49.373992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 12 17:20:49.373992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 12 17:20:49.373992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 12 17:20:49.373992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application result = self._query(query) Jul 12 17:20:49.373992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 12 17:20:49.373992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application conn.query(q) Jul 12 17:20:49.373992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 12 17:20:49.373992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 12 17:20:49.373992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 12 17:20:49.373992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application result.read() Jul 12 17:20:49.373992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 12 17:20:49.373992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 12 17:20:49.373992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 12 17:20:49.373992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 12 17:20:49.376437 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 12 17:20:49.376437 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 12 17:20:49.376437 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 12 17:20:49.376437 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 12 17:20:49.376437 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Jul 12 17:20:49.376437 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Jul 12 17:20:49.376437 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [parameters: {'user_id': '1a81453c45404c5da9c085e0e7e2e949', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Jul 12 17:20:49.376437 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Jul 12 17:20:49.376437 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:49.376437 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 12 17:20:49.376437 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:49.376437 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:49.376437 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:49.376437 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:49.376437 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:49.378590 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:49.378590 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:49.378590 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:49.378590 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:49.378590 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:49.378590 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:49.378590 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:49.378590 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Jul 12 17:20:49.378590 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Jul 12 17:20:49.378590 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:49.378590 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:49.378590 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:20:49.378590 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:49.378590 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:20:49.378590 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:49.378590 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Jul 12 17:20:49.378590 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Jul 12 17:20:49.380383 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Jul 12 17:20:49.380383 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Jul 12 17:20:49.380383 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Jul 12 17:20:49.380383 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Jul 12 17:20:49.380383 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Jul 12 17:20:49.380383 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:49.380383 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 488/973] 10.222.0.16 () {60 vars in 1131 bytes} [Tue Jul 12 17:20:49 2022] POST /identity/v3/users => generated 160 bytes in 268 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:20:49.380383 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64af076a-3baf-4d27-bf3b-e4a7c6d8c70d None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:49.393474 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a63595-7fe8-40e2-a3e8-8801851e3b03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:49.394846 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a63595-7fe8-40e2-a3e8-8801851e3b03 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:49.395812 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a63595-7fe8-40e2-a3e8-8801851e3b03 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:49.396254 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a63595-7fe8-40e2-a3e8-8801851e3b03 None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:49.397472 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a63595-7fe8-40e2-a3e8-8801851e3b03 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:49.397954 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a63595-7fe8-40e2-a3e8-8801851e3b03 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:49.400183 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a63595-7fe8-40e2-a3e8-8801851e3b03 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:49.416678 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 486/974] 10.222.0.16 () {56 vars in 1115 bytes} [Tue Jul 12 17:20:49 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:49.417312 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a63595-7fe8-40e2-a3e8-8801851e3b03 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:49.449911 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:49.515531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64af076a-3baf-4d27-bf3b-e4a7c6d8c70d None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:49.521131 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:49.549386 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a63595-7fe8-40e2-a3e8-8801851e3b03 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:49.560697 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c99d01c7-7340-44db-841d-bb1bbcfb0c53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:49.561257 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99d01c7-7340-44db-841d-bb1bbcfb0c53 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:49.561524 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99d01c7-7340-44db-841d-bb1bbcfb0c53 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:49.561808 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99d01c7-7340-44db-841d-bb1bbcfb0c53 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:49.582068 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c99d01c7-7340-44db-841d-bb1bbcfb0c53 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:49.583888 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c99d01c7-7340-44db-841d-bb1bbcfb0c53 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:49.597363 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bde2551-482d-406e-8f7d-06840d08ff11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:49.598194 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bde2551-482d-406e-8f7d-06840d08ff11 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:49.598592 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bde2551-482d-406e-8f7d-06840d08ff11 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:49.598945 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bde2551-482d-406e-8f7d-06840d08ff11 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:49.599472 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-c99d01c7-7340-44db-841d-bb1bbcfb0c53 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jul 12 17:20:49.599472 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:49.599472 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:49.599472 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:49.599472 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:49.599472 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:49.599472 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:49.599472 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:49.599472 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:49.599472 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:49.599472 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:49.599472 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:49.599472 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:20:49.599472 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:20:49.599472 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:20:49.599472 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:20:49.600820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:49.600820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:49.600820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:20:49.600820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:49.600820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:20:49.600820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:49.600820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:49.600820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:49.600820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:49.600820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:49.600820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:49.600820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:49.600820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:49.600820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:49.600820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:49.600820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:49.600820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:49.600820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:49.601823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:49.601823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:49.601823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:20:49.601823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:20:49.601823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:20:49.601823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:20:49.601823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:20:49.601823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:20:49.601823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jul 12 17:20:49.601823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:49.601823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 489/975] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:20:49 2022] GET /identity/v3/users/cinder => generated 84 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:49.602878 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22e0338-8959-4f22-a1e9-76cb7f074392 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:49.602878 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22e0338-8959-4f22-a1e9-76cb7f074392 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:49.603021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22e0338-8959-4f22-a1e9-76cb7f074392 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:49.610468 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 490/976] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:49 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:49.610468 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a22e0338-8959-4f22-a1e9-76cb7f074392 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:49.621524 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bde2551-482d-406e-8f7d-06840d08ff11 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:49.623054 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bde2551-482d-406e-8f7d-06840d08ff11 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:49.642012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-3bde2551-482d-406e-8f7d-06840d08ff11 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:49.642012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:49.642012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:49.642012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:49.642012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:49.642012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:49.642012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:49.642012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:49.642012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:49.642012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:49.642012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:49.642012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:49.642012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:49.642012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:49.642012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:49.643018 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:49.643018 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:49.643018 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:49.643018 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:49.643018 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:49.643018 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:49.643018 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:49.643018 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:49.643018 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:49.643018 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:49.643018 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:49.643018 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:49.643018 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:49.643018 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:49.643018 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:49.643018 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:49.643018 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:49.643929 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:49.643929 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:49.643929 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:49.643929 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:49.643929 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 487/977] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:20:49.644752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67dc4a41-fdc7-436e-a3de-c87ff772feda None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:49.644752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67dc4a41-fdc7-436e-a3de-c87ff772feda None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:49.645018 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67dc4a41-fdc7-436e-a3de-c87ff772feda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:49.664992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-67dc4a41-fdc7-436e-a3de-c87ff772feda None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:49.666185 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-67dc4a41-fdc7-436e-a3de-c87ff772feda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:49.728131 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a22e0338-8959-4f22-a1e9-76cb7f074392 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:49.735653 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 488/978] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:49 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:20:49.736682 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67dc4a41-fdc7-436e-a3de-c87ff772feda None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:49.747921 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:49.776619 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d99f84f9-c653-4f63-b931-42473e143904 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:49.778211 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99f84f9-c653-4f63-b931-42473e143904 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:49.778536 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99f84f9-c653-4f63-b931-42473e143904 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:49.778829 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99f84f9-c653-4f63-b931-42473e143904 None admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:49.779643 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d99f84f9-c653-4f63-b931-42473e143904 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:49.779956 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d99f84f9-c653-4f63-b931-42473e143904 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:49.781413 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d99f84f9-c653-4f63-b931-42473e143904 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:49.802424 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:49.826467 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 491/979] 10.222.0.16 () {56 vars in 1101 bytes} [Tue Jul 12 17:20:49 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:49.826467 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d99f84f9-c653-4f63-b931-42473e143904 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:49.871248 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67dc4a41-fdc7-436e-a3de-c87ff772feda None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:49.910240 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77bfcece-bde7-4795-a3c7-85d8801e311f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:49.911163 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bfcece-bde7-4795-a3c7-85d8801e311f None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:49.911558 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bfcece-bde7-4795-a3c7-85d8801e311f None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:49.911971 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bfcece-bde7-4795-a3c7-85d8801e311f None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:49.912892 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77bfcece-bde7-4795-a3c7-85d8801e311f None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:49.913313 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77bfcece-bde7-4795-a3c7-85d8801e311f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:49.914938 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77bfcece-bde7-4795-a3c7-85d8801e311f None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:49.934294 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 489/980] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:49.935667 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77bfcece-bde7-4795-a3c7-85d8801e311f None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:49.962637 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d99f84f9-c653-4f63-b931-42473e143904 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:50.011919 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5198f3-b2e9-45e6-960a-50b3fb94af3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:50.012638 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5198f3-b2e9-45e6-960a-50b3fb94af3c None admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:50.012929 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5198f3-b2e9-45e6-960a-50b3fb94af3c None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:50.013210 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5198f3-b2e9-45e6-960a-50b3fb94af3c None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:50.013944 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f5198f3-b2e9-45e6-960a-50b3fb94af3c None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:50.016067 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f5198f3-b2e9-45e6-960a-50b3fb94af3c None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:50.060249 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77bfcece-bde7-4795-a3c7-85d8801e311f None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:50.083840 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:50.098384 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96eb5eb4-d1de-43ff-a3b0-589186ce45e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:50.099248 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96eb5eb4-d1de-43ff-a3b0-589186ce45e3 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:50.099542 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96eb5eb4-d1de-43ff-a3b0-589186ce45e3 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:50.099823 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96eb5eb4-d1de-43ff-a3b0-589186ce45e3 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:50.109742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 492/981] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Jul 12 17:20:49 2022] POST /identity/v3/services => generated 242 bytes in 290 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:20:50.109899 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96eb5eb4-d1de-43ff-a3b0-589186ce45e3 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:50.111459 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96eb5eb4-d1de-43ff-a3b0-589186ce45e3 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:50.128370 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-96eb5eb4-d1de-43ff-a3b0-589186ce45e3 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 12 17:20:50.128370 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:50.128370 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:50.128370 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:50.128370 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:50.128370 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:50.128370 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:50.128370 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:50.128370 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:50.128370 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:50.128370 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:50.128370 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:50.128370 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:20:50.128370 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:20:50.128370 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:20:50.128370 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:20:50.129764 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:50.129764 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:50.129764 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:20:50.129764 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:50.129764 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:20:50.129764 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:50.129764 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:50.129764 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:50.129764 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:50.129764 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:50.129764 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:50.129764 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:50.129764 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:50.129764 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:50.129764 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:50.129764 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:50.129764 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:50.129764 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:50.134684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:50.134684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:50.134684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:20:50.134684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:20:50.134684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:20:50.134684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:20:50.134684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:20:50.134684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:20:50.134684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 12 17:20:50.134684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:50.134684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 490/982] 10.222.0.16 () {56 vars in 1117 bytes} [Tue Jul 12 17:20:49 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:50.135348 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5198f3-b2e9-45e6-960a-50b3fb94af3c None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:50.167910 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:50.259888 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5198f3-b2e9-45e6-960a-50b3fb94af3c None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:50.297745 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2cd91f6-5a46-4c8f-8130-afb35d0a9bf8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:50.298489 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2cd91f6-5a46-4c8f-8130-afb35d0a9bf8 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:50.298813 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2cd91f6-5a46-4c8f-8130-afb35d0a9bf8 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:50.299098 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2cd91f6-5a46-4c8f-8130-afb35d0a9bf8 None admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:50.299942 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2cd91f6-5a46-4c8f-8130-afb35d0a9bf8 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:50.300258 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2cd91f6-5a46-4c8f-8130-afb35d0a9bf8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:50.301683 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2cd91f6-5a46-4c8f-8130-afb35d0a9bf8 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:50.320570 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 493/983] 10.222.0.16 () {56 vars in 1149 bytes} [Tue Jul 12 17:20:50 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:20:50.326785 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96eb5eb4-d1de-43ff-a3b0-589186ce45e3 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:50.429776 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96eb5eb4-d1de-43ff-a3b0-589186ce45e3 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:50.467055 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4c826c-6e73-4bbd-85f8-7428ce26dff6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:50.467903 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4c826c-6e73-4bbd-85f8-7428ce26dff6 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:50.468025 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4c826c-6e73-4bbd-85f8-7428ce26dff6 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:50.468274 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4c826c-6e73-4bbd-85f8-7428ce26dff6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:50.518825 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:50.539148 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d4c826c-6e73-4bbd-85f8-7428ce26dff6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:50.540744 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d4c826c-6e73-4bbd-85f8-7428ce26dff6 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:50.668669 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 491/984] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:20:50 2022] GET /identity/v3/auth/tokens => generated 1560 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 12 17:20:50.670225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2cd91f6-5a46-4c8f-8130-afb35d0a9bf8 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:50.753002 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:50.771520 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2cd91f6-5a46-4c8f-8130-afb35d0a9bf8 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:50.811162 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c23cf9-1658-43a9-b5d0-df7d8a46b883 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:50.812056 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c23cf9-1658-43a9-b5d0-df7d8a46b883 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:50.812415 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c23cf9-1658-43a9-b5d0-df7d8a46b883 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:50.812765 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c23cf9-1658-43a9-b5d0-df7d8a46b883 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:50.823410 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c23cf9-1658-43a9-b5d0-df7d8a46b883 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:50.825455 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c23cf9-1658-43a9-b5d0-df7d8a46b883 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:50.844781 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-e3c23cf9-1658-43a9-b5d0-df7d8a46b883 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:50.844781 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:50.844781 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:50.844781 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:50.844781 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:50.844781 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:50.844781 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:50.844781 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:50.844781 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:50.844781 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:50.844781 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:50.844781 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:50.844781 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:50.844781 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:50.844781 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:50.845981 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:50.845981 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:50.845981 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:50.845981 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:50.845981 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:50.845981 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:50.845981 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:50.845981 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:50.845981 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:50.845981 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:50.845981 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:50.845981 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:50.845981 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:50.845981 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:50.845981 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:50.845981 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:50.845981 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:50.847188 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:50.847188 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:50.847188 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:50.847188 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:50.852343 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 494/985] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:50.853655 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4c826c-6e73-4bbd-85f8-7428ce26dff6 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:50.937252 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:50.957266 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4c826c-6e73-4bbd-85f8-7428ce26dff6 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:50.993819 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a476301b-fdb5-4bff-92c9-3ee714d54af6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:50.994528 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a476301b-fdb5-4bff-92c9-3ee714d54af6 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:50.994816 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a476301b-fdb5-4bff-92c9-3ee714d54af6 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:50.995095 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a476301b-fdb5-4bff-92c9-3ee714d54af6 None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:50.996449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a476301b-fdb5-4bff-92c9-3ee714d54af6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:50.996760 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a476301b-fdb5-4bff-92c9-3ee714d54af6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:50.998183 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a476301b-fdb5-4bff-92c9-3ee714d54af6 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:51.025475 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70370234-d85b-4c8e-addc-73f6fb7a93a8 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:51.025699 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70370234-d85b-4c8e-addc-73f6fb7a93a8 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:51.025985 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70370234-d85b-4c8e-addc-73f6fb7a93a8 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:51.027490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 495/986] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 12 17:20:51.032591 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d940db8-0a36-433f-8cfd-7aebc1dbe020 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:51.032885 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d940db8-0a36-433f-8cfd-7aebc1dbe020 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:51.033171 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d940db8-0a36-433f-8cfd-7aebc1dbe020 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:51.036927 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 492/987] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:51.075334 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a476301b-fdb5-4bff-92c9-3ee714d54af6 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:51.076055 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-6d940db8-0a36-433f-8cfd-7aebc1dbe020 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:51.076055 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-6d940db8-0a36-433f-8cfd-7aebc1dbe020 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:51.142568 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 496/988] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:51 2022] POST /identity/v3/auth/tokens => generated 1560 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 12 17:20:51.166344 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d940db8-0a36-433f-8cfd-7aebc1dbe020 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:51.170907 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a476301b-fdb5-4bff-92c9-3ee714d54af6 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:51.211359 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb16749-8023-4846-9b2c-aa60da373a8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:51.212166 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb16749-8023-4846-9b2c-aa60da373a8c None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:51.212458 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb16749-8023-4846-9b2c-aa60da373a8c None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:51.212741 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb16749-8023-4846-9b2c-aa60da373a8c None admin] PATH_INFO: `/v3/projects/service` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:51.220789 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb16749-8023-4846-9b2c-aa60da373a8c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:51.222461 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb16749-8023-4846-9b2c-aa60da373a8c None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:51.225917 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:51.239515 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-fbb16749-8023-4846-9b2c-aa60da373a8c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:51.239515 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:51.239515 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:51.239515 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:51.239515 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:51.239515 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:51.239515 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:51.239515 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:51.239515 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:51.239515 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:51.239515 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:51.239515 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:51.239515 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:20:51.239515 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:20:51.239515 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:20:51.241900 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:20:51.241900 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:51.241900 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:51.241900 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:51.241900 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:51.241900 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:51.241900 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:51.241900 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:51.241900 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:51.241900 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:51.241900 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:51.241900 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:51.241900 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:51.241900 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:51.241900 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:51.241900 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:51.241900 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:51.242918 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:20:51.242918 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:20:51.242918 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:20:51.242918 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:20:51.242918 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:20:51.242918 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:20:51.242918 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:20:51.242918 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:51.242918 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 493/989] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:20:51 2022] GET /identity/v3/projects/service => generated 88 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:20:51.243424 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:51.246860 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb16749-8023-4846-9b2c-aa60da373a8c None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:51.286677 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d940db8-0a36-433f-8cfd-7aebc1dbe020 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:51.325415 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b5719e-6c12-416f-95a5-23e7d9aa3259 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:51.326153 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b5719e-6c12-416f-95a5-23e7d9aa3259 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:51.326466 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b5719e-6c12-416f-95a5-23e7d9aa3259 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:51.326751 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b5719e-6c12-416f-95a5-23e7d9aa3259 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:51.327415 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b5719e-6c12-416f-95a5-23e7d9aa3259 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:51.328810 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b5719e-6c12-416f-95a5-23e7d9aa3259 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:51.341783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-f8b5719e-6c12-416f-95a5-23e7d9aa3259 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 12 17:20:51.341783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:51.341783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 12 17:20:51.341783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 12 17:20:51.341783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 12 17:20:51.341783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 12 17:20:51.341783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 12 17:20:51.341783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:51.341783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 12 17:20:51.341783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:51.341783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 12 17:20:51.341783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:51.341783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:51.341783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:51.341783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:51.341783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:51.341783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:51.342864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:51.342864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:51.342864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:51.342864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:51.342864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:51.342864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:51.342864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 12 17:20:51.342864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 12 17:20:51.342864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 12 17:20:51.342864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 12 17:20:51.342864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:51.342864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:51.342864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:51.342864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:51.342864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:51.342864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:51.347469 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:51.347469 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:51.347469 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:51.347469 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:51.347469 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:51.347469 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:51.347469 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:51.347469 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:51.347469 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:51.347469 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:51.347469 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 12 17:20:51.347469 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:51.347469 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 12 17:20:51.347469 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:51.347469 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 497/990] 10.222.0.16 () {56 vars in 1126 bytes} [Tue Jul 12 17:20:51 2022] GET /identity/v3/services/object-store => generated 93 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:20:51.349357 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b5719e-6c12-416f-95a5-23e7d9aa3259 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:51.361246 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb16749-8023-4846-9b2c-aa60da373a8c None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:51.396984 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b9232d-8bb1-4121-a386-fcb1d9ce7112 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:51.397651 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b9232d-8bb1-4121-a386-fcb1d9ce7112 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:51.397936 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b9232d-8bb1-4121-a386-fcb1d9ce7112 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:51.398216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b9232d-8bb1-4121-a386-fcb1d9ce7112 None admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:51.399035 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b9232d-8bb1-4121-a386-fcb1d9ce7112 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:51.399345 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b9232d-8bb1-4121-a386-fcb1d9ce7112 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:51.400762 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b9232d-8bb1-4121-a386-fcb1d9ce7112 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:51.413600 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 494/991] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Jul 12 17:20:51 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:20:51.419774 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b9232d-8bb1-4121-a386-fcb1d9ce7112 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:51.462991 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b5719e-6c12-416f-95a5-23e7d9aa3259 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:51.481744 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:51.496684 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:51.505569 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f0d21e-bf72-43cc-b780-013ad14c211d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:51.506268 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f0d21e-bf72-43cc-b780-013ad14c211d None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:51.506557 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f0d21e-bf72-43cc-b780-013ad14c211d None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:51.506837 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f0d21e-bf72-43cc-b780-013ad14c211d None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:51.507548 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f0d21e-bf72-43cc-b780-013ad14c211d None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:51.507859 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f0d21e-bf72-43cc-b780-013ad14c211d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:51.509255 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f0d21e-bf72-43cc-b780-013ad14c211d None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:51.515014 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 498/992] 10.222.0.16 () {56 vars in 1122 bytes} [Tue Jul 12 17:20:51 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:51.520624 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f0d21e-bf72-43cc-b780-013ad14c211d None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:51.528268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b9232d-8bb1-4121-a386-fcb1d9ce7112 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:51.569618 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9788369c-3717-40f9-871e-1b552e80f165 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:51.570211 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9788369c-3717-40f9-871e-1b552e80f165 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:51.570526 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9788369c-3717-40f9-871e-1b552e80f165 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:51.570813 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9788369c-3717-40f9-871e-1b552e80f165 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:51.571659 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9788369c-3717-40f9-871e-1b552e80f165 None admin] RBAC: Adding query filter params (role.id=feb6a96f7b6c4f0a8d64be226f85ee6c, scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2, user.id=3ac0d5d8de4943a3888abee3b24b8a6f) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:51.571978 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9788369c-3717-40f9-871e-1b552e80f165 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:51.573712 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9788369c-3717-40f9-871e-1b552e80f165 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:51.580088 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 495/993] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:20:51 2022] GET /identity/v3/role_assignments?user.id=3ac0d5d8de4943a3888abee3b24b8a6f&scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2&role.id=feb6a96f7b6c4f0a8d64be226f85ee6c => generated 608 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:20:51.644211 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f0d21e-bf72-43cc-b780-013ad14c211d None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:51.686844 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb8dacf-8eaf-424d-8715-ee6837c0b06c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:51.687600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb8dacf-8eaf-424d-8715-ee6837c0b06c None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:51.687816 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb8dacf-8eaf-424d-8715-ee6837c0b06c None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:51.688097 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb8dacf-8eaf-424d-8715-ee6837c0b06c None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:51.688885 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb8dacf-8eaf-424d-8715-ee6837c0b06c None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:51.689233 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb8dacf-8eaf-424d-8715-ee6837c0b06c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:51.690733 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb8dacf-8eaf-424d-8715-ee6837c0b06c None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:51.701384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 499/994] 10.222.0.16 () {56 vars in 1122 bytes} [Tue Jul 12 17:20:51 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:20:51.705068 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9788369c-3717-40f9-871e-1b552e80f165 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:51.853855 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9788369c-3717-40f9-871e-1b552e80f165 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:51.878700 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:51.890603 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12cda9e7-a067-4765-8b3f-738e4401d696 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:51.891285 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cda9e7-a067-4765-8b3f-738e4401d696 None admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:51.896929 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cda9e7-a067-4765-8b3f-738e4401d696 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:51.896929 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cda9e7-a067-4765-8b3f-738e4401d696 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:51.896929 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12cda9e7-a067-4765-8b3f-738e4401d696 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:51.896929 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12cda9e7-a067-4765-8b3f-738e4401d696 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:51.986052 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 496/995] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Jul 12 17:20:51 2022] POST /identity/v3/endpoints => generated 325 bytes in 276 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:20:52.222784 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb7bf9a-8607-457b-9685-5249c8e9c265 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:52.223176 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb7bf9a-8607-457b-9685-5249c8e9c265 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:52.223532 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb7bf9a-8607-457b-9685-5249c8e9c265 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:52.225487 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 500/996] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:52.230924 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6304ea0c-bde1-42b6-beea-358927453e45 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:52.231228 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6304ea0c-bde1-42b6-beea-358927453e45 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:52.231525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6304ea0c-bde1-42b6-beea-358927453e45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:52.248644 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-6304ea0c-bde1-42b6-beea-358927453e45 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:52.249845 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-6304ea0c-bde1-42b6-beea-358927453e45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:52.326549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 497/997] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:52 2022] POST /identity/v3/auth/tokens => generated 1707 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:20:52.349687 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb7bf9a-8607-457b-9685-5249c8e9c265 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:52.412811 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:52.460673 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb7bf9a-8607-457b-9685-5249c8e9c265 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:52.496066 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb673fa1-16ae-4cea-aa88-e2ed36f7a707 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:52.496811 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb673fa1-16ae-4cea-aa88-e2ed36f7a707 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:52.497101 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb673fa1-16ae-4cea-aa88-e2ed36f7a707 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:52.497380 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb673fa1-16ae-4cea-aa88-e2ed36f7a707 None admin] PATH_INFO: `/v3/services/compute` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:52.498139 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb673fa1-16ae-4cea-aa88-e2ed36f7a707 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:52.499552 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb673fa1-16ae-4cea-aa88-e2ed36f7a707 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:52.512439 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-bb673fa1-16ae-4cea-aa88-e2ed36f7a707 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Jul 12 17:20:52.512439 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:52.512439 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 12 17:20:52.512439 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 12 17:20:52.512439 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 12 17:20:52.512439 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 12 17:20:52.512439 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 12 17:20:52.512439 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:52.512439 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 12 17:20:52.512439 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:52.512439 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 12 17:20:52.512439 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:52.512439 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:52.512439 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:52.512439 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:52.512439 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:52.512439 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:52.513454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:52.513454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:52.513454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:52.513454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:52.513454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:52.513454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:52.513454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 12 17:20:52.513454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 12 17:20:52.513454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 12 17:20:52.513454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 12 17:20:52.513454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:52.513454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:52.513454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:52.513454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:52.513454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:52.513454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:52.514337 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:52.514337 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:52.514337 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:52.514337 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:52.514337 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:52.514337 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:52.514337 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:52.514337 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:52.514337 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:52.514337 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:52.514337 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 12 17:20:52.514337 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:52.514337 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 12 17:20:52.514337 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:52.518809 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 501/998] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:20:52 2022] GET /identity/v3/services/compute => generated 88 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:52.519617 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6304ea0c-bde1-42b6-beea-358927453e45 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:52.631209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6304ea0c-bde1-42b6-beea-358927453e45 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:52.643190 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:52.668801 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e058edb0-dbc6-47f0-8f51-5ad8b492e0c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:52.669731 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e058edb0-dbc6-47f0-8f51-5ad8b492e0c3 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:52.670019 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e058edb0-dbc6-47f0-8f51-5ad8b492e0c3 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:52.670358 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e058edb0-dbc6-47f0-8f51-5ad8b492e0c3 None admin] PATH_INFO: `/v3/services` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:52.671054 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e058edb0-dbc6-47f0-8f51-5ad8b492e0c3 None admin] RBAC: Adding query filter params (name=compute) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:52.671377 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e058edb0-dbc6-47f0-8f51-5ad8b492e0c3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:52.672727 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e058edb0-dbc6-47f0-8f51-5ad8b492e0c3 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:52.682351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 498/999] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Jul 12 17:20:52 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:52.683406 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb673fa1-16ae-4cea-aa88-e2ed36f7a707 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:52.787763 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb673fa1-16ae-4cea-aa88-e2ed36f7a707 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:52.810831 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:52.827768 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2dc17c-70d4-4bdc-acec-5e7323be5895 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:52.828441 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2dc17c-70d4-4bdc-acec-5e7323be5895 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:52.828731 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2dc17c-70d4-4bdc-acec-5e7323be5895 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:52.829012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2dc17c-70d4-4bdc-acec-5e7323be5895 None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:52.829722 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b2dc17c-70d4-4bdc-acec-5e7323be5895 None admin] RBAC: Adding query filter params (type=compute) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:52.830035 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b2dc17c-70d4-4bdc-acec-5e7323be5895 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:52.831568 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b2dc17c-70d4-4bdc-acec-5e7323be5895 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:52.836373 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 502/1000] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Jul 12 17:20:52 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:52.842718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e058edb0-dbc6-47f0-8f51-5ad8b492e0c3 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:52.946928 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e058edb0-dbc6-47f0-8f51-5ad8b492e0c3 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:52.983350 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f067ff74-6d3f-4ad6-827e-922cfc46189f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:52.984014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f067ff74-6d3f-4ad6-827e-922cfc46189f None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:52.984304 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f067ff74-6d3f-4ad6-827e-922cfc46189f None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:52.984587 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f067ff74-6d3f-4ad6-827e-922cfc46189f None admin] PATH_INFO: `/v3/endpoints` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:52.985378 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f067ff74-6d3f-4ad6-827e-922cfc46189f None admin] RBAC: Adding query filter params (interface=public, service_id=468d47744cfc47eea09182222952e28b, region_id=RegionOne) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:52.985696 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f067ff74-6d3f-4ad6-827e-922cfc46189f None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:52.987115 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f067ff74-6d3f-4ad6-827e-922cfc46189f None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:52.992442 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 499/1001] 10.222.0.16 () {56 vars in 1251 bytes} [Tue Jul 12 17:20:52 2022] GET /identity/v3/endpoints?service_id=468d47744cfc47eea09182222952e28b&interface=public®ion_id=RegionOne => generated 194 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:52.998027 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2dc17c-70d4-4bdc-acec-5e7323be5895 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:53.103628 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2dc17c-70d4-4bdc-acec-5e7323be5895 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:53.139940 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c3788f-fb2f-47e2-8bcf-6fb8b2998269 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:53.140613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c3788f-fb2f-47e2-8bcf-6fb8b2998269 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:53.140902 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c3788f-fb2f-47e2-8bcf-6fb8b2998269 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:53.141185 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c3788f-fb2f-47e2-8bcf-6fb8b2998269 None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:53.141878 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c3788f-fb2f-47e2-8bcf-6fb8b2998269 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:53.143262 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c3788f-fb2f-47e2-8bcf-6fb8b2998269 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:53.150088 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 503/1002] 10.222.0.16 () {56 vars in 1087 bytes} [Tue Jul 12 17:20:52 2022] GET /identity/v3/services => generated 1727 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:20:53.722209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fba1130-6834-4930-9af3-35d934250157 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:53.722788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fba1130-6834-4930-9af3-35d934250157 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:53.722877 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fba1130-6834-4930-9af3-35d934250157 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:53.724237 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 500/1003] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:53.729963 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e133ab0-b6d2-4210-96b5-bf07624ace8d None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:53.730261 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e133ab0-b6d2-4210-96b5-bf07624ace8d None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:53.730567 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e133ab0-b6d2-4210-96b5-bf07624ace8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:53.747722 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-0e133ab0-b6d2-4210-96b5-bf07624ace8d None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:53.748899 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-0e133ab0-b6d2-4210-96b5-bf07624ace8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:53.795139 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:53.820135 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 504/1004] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:53 2022] POST /identity/v3/auth/tokens => generated 1707 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:20:53.842190 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fba1130-6834-4930-9af3-35d934250157 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:53.887177 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:53.951522 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fba1130-6834-4930-9af3-35d934250157 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:53.988787 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1659976-b843-4dcb-b728-a10709c5b4fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:53.989582 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1659976-b843-4dcb-b728-a10709c5b4fb None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:53.989870 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1659976-b843-4dcb-b728-a10709c5b4fb None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:53.990152 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1659976-b843-4dcb-b728-a10709c5b4fb None admin] PATH_INFO: `/v3/services/image` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:53.990853 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1659976-b843-4dcb-b728-a10709c5b4fb None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:53.992341 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1659976-b843-4dcb-b728-a10709c5b4fb None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:54.004940 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-a1659976-b843-4dcb-b728-a10709c5b4fb None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Jul 12 17:20:54.004940 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:54.004940 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 12 17:20:54.004940 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 12 17:20:54.004940 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 12 17:20:54.004940 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 12 17:20:54.004940 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 12 17:20:54.004940 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:54.004940 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 12 17:20:54.004940 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:54.004940 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 12 17:20:54.004940 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:54.004940 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:54.004940 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:54.004940 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:54.004940 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:54.004940 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:54.006012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:54.006012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:54.006012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:54.006012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:54.006012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:54.006012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:54.006012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 12 17:20:54.006012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 12 17:20:54.006012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 12 17:20:54.006012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 12 17:20:54.006012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:54.006012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:54.006012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:54.006012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:54.006012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:54.006012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:54.007046 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:54.007046 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:54.007046 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:54.007046 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:54.007046 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:54.007046 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:54.007046 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:54.007046 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:54.007046 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:54.007046 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:54.007046 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 12 17:20:54.007046 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:54.007046 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 12 17:20:54.007046 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:54.007046 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 501/1005] 10.222.0.16 () {56 vars in 1105 bytes} [Tue Jul 12 17:20:53 2022] GET /identity/v3/services/image => generated 86 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:54.012770 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e133ab0-b6d2-4210-96b5-bf07624ace8d None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:54.106265 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8375c4f-9cb4-4952-8255-dd71ccdf7f34 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:54.106609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8375c4f-9cb4-4952-8255-dd71ccdf7f34 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:54.106796 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8375c4f-9cb4-4952-8255-dd71ccdf7f34 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:54.108198 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 502/1006] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 12 17:20:54.113603 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337abede-e894-4e49-ad44-c48415367410 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:54.113900 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337abede-e894-4e49-ad44-c48415367410 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:54.114182 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337abede-e894-4e49-ad44-c48415367410 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:54.114943 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e133ab0-b6d2-4210-96b5-bf07624ace8d None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:54.131363 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-337abede-e894-4e49-ad44-c48415367410 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:54.132563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-337abede-e894-4e49-ad44-c48415367410 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:54.152059 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef7e1d1-2fa3-446a-9b3e-db8e88b18394 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:54.152725 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef7e1d1-2fa3-446a-9b3e-db8e88b18394 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:54.153057 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef7e1d1-2fa3-446a-9b3e-db8e88b18394 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:54.153340 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef7e1d1-2fa3-446a-9b3e-db8e88b18394 None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:54.154040 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef7e1d1-2fa3-446a-9b3e-db8e88b18394 None admin] RBAC: Adding query filter params (name=image) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:54.154352 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef7e1d1-2fa3-446a-9b3e-db8e88b18394 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:54.155761 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef7e1d1-2fa3-446a-9b3e-db8e88b18394 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:54.160957 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 505/1007] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:54 2022] GET /identity/v3/services?name=image => generated 122 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:20:54.166061 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef7e1d1-2fa3-446a-9b3e-db8e88b18394 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:54.214490 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 503/1008] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:54 2022] POST /identity/v3/auth/tokens => generated 1707 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 12 17:20:54.218769 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:54.234713 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337abede-e894-4e49-ad44-c48415367410 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:54.269339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:54.289146 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef7e1d1-2fa3-446a-9b3e-db8e88b18394 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:54.327889 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e42a48-ef04-4ada-a243-8a50ecd8a4c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:54.328563 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e42a48-ef04-4ada-a243-8a50ecd8a4c3 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:54.328851 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e42a48-ef04-4ada-a243-8a50ecd8a4c3 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:54.329132 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e42a48-ef04-4ada-a243-8a50ecd8a4c3 None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:54.329846 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e42a48-ef04-4ada-a243-8a50ecd8a4c3 None admin] RBAC: Adding query filter params (type=image) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:54.330165 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e42a48-ef04-4ada-a243-8a50ecd8a4c3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:54.331569 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e42a48-ef04-4ada-a243-8a50ecd8a4c3 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:54.342836 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 506/1009] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:54 2022] GET /identity/v3/services?type=image => generated 122 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:20:54.345898 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337abede-e894-4e49-ad44-c48415367410 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:54.387428 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62503a9-9656-434e-af08-0b869032e1a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:54.388170 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62503a9-9656-434e-af08-0b869032e1a6 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:54.388456 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62503a9-9656-434e-af08-0b869032e1a6 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:54.388859 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62503a9-9656-434e-af08-0b869032e1a6 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:54.390168 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62503a9-9656-434e-af08-0b869032e1a6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:54.391714 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62503a9-9656-434e-af08-0b869032e1a6 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:54.393722 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 504/1010] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:54 2022] GET /identity/v3/domains/default => generated 199 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:20:54.399843 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e42a48-ef04-4ada-a243-8a50ecd8a4c3 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:54.515830 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e42a48-ef04-4ada-a243-8a50ecd8a4c3 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:54.525001 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:54.553510 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2088a1d-7651-40c2-b96a-7655c2852728 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:54.554174 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2088a1d-7651-40c2-b96a-7655c2852728 None admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:54.554526 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2088a1d-7651-40c2-b96a-7655c2852728 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:54.554807 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2088a1d-7651-40c2-b96a-7655c2852728 None admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:54.555523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2088a1d-7651-40c2-b96a-7655c2852728 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:54.556940 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2088a1d-7651-40c2-b96a-7655c2852728 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:54.656081 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 507/1011] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Jul 12 17:20:54 2022] POST /identity/v3/projects => generated 311 bytes in 257 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:20:55.193976 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1508a642-fa68-40e9-a33d-2e731ce9dc36 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:55.194521 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1508a642-fa68-40e9-a33d-2e731ce9dc36 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:55.194640 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1508a642-fa68-40e9-a33d-2e731ce9dc36 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:55.196165 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 505/1012] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:55.201955 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bf8d9e-270a-473d-b3b4-4572235eaa27 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:55.202451 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bf8d9e-270a-473d-b3b4-4572235eaa27 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:55.202534 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bf8d9e-270a-473d-b3b4-4572235eaa27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:55.220061 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-60bf8d9e-270a-473d-b3b4-4572235eaa27 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:55.221249 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-60bf8d9e-270a-473d-b3b4-4572235eaa27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:55.292889 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 508/1013] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:55 2022] POST /identity/v3/auth/tokens => generated 1707 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:20:55.315136 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1508a642-fa68-40e9-a33d-2e731ce9dc36 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:55.418753 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:55.423617 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1508a642-fa68-40e9-a33d-2e731ce9dc36 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:55.461653 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bcb516e-8be5-4ef7-a89a-edd544999625 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:55.462535 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcb516e-8be5-4ef7-a89a-edd544999625 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:55.462848 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcb516e-8be5-4ef7-a89a-edd544999625 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:55.463126 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcb516e-8be5-4ef7-a89a-edd544999625 None admin] PATH_INFO: `/v3/services/compute` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:55.463909 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bcb516e-8be5-4ef7-a89a-edd544999625 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:55.465470 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bcb516e-8be5-4ef7-a89a-edd544999625 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:55.477992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-7bcb516e-8be5-4ef7-a89a-edd544999625 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Jul 12 17:20:55.477992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:55.477992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 12 17:20:55.477992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 12 17:20:55.477992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 12 17:20:55.477992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 12 17:20:55.477992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 12 17:20:55.477992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:55.477992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 12 17:20:55.477992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:55.477992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 12 17:20:55.477992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:55.477992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:55.477992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:55.477992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:55.477992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:55.477992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:55.479097 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:55.479097 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:55.479097 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:55.479097 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:55.479097 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:55.479097 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:55.479097 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 12 17:20:55.479097 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 12 17:20:55.479097 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 12 17:20:55.479097 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 12 17:20:55.479097 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:55.479097 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:55.479097 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:55.479097 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:55.479097 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:55.479097 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:55.480007 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:55.480007 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:55.480007 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:55.480007 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:55.480007 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:55.480007 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:55.480007 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:55.480007 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:55.480007 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:55.480007 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:55.480007 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 12 17:20:55.480007 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:55.480007 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 12 17:20:55.480007 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:55.482942 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 506/1014] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:20:55 2022] GET /identity/v3/services/compute => generated 88 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:55.485902 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60bf8d9e-270a-473d-b3b4-4572235eaa27 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:55.584204 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:55.595958 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60bf8d9e-270a-473d-b3b4-4572235eaa27 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:55.631527 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32d2cde-d29f-496e-a911-9e0566f20288 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:55.632204 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32d2cde-d29f-496e-a911-9e0566f20288 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:55.636409 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32d2cde-d29f-496e-a911-9e0566f20288 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:55.636409 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32d2cde-d29f-496e-a911-9e0566f20288 None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:55.636409 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b32d2cde-d29f-496e-a911-9e0566f20288 None admin] RBAC: Adding query filter params (name=compute) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:55.636409 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b32d2cde-d29f-496e-a911-9e0566f20288 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:55.636409 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b32d2cde-d29f-496e-a911-9e0566f20288 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:55.645980 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 509/1015] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Jul 12 17:20:55 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:55.647282 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bcb516e-8be5-4ef7-a89a-edd544999625 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:55.749768 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bcb516e-8be5-4ef7-a89a-edd544999625 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:55.784292 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d42f0886-8a55-4473-998c-b33f767f0808 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:55.785337 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42f0886-8a55-4473-998c-b33f767f0808 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:55.785854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42f0886-8a55-4473-998c-b33f767f0808 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:55.786353 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42f0886-8a55-4473-998c-b33f767f0808 None admin] PATH_INFO: `/v3/services` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:55.787314 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d42f0886-8a55-4473-998c-b33f767f0808 None admin] RBAC: Adding query filter params (type=compute) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:55.787854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d42f0886-8a55-4473-998c-b33f767f0808 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:55.789440 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d42f0886-8a55-4473-998c-b33f767f0808 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:55.799808 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 507/1016] 10.222.0.16 () {56 vars in 1112 bytes} [Tue Jul 12 17:20:55 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:20:55.801029 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32d2cde-d29f-496e-a911-9e0566f20288 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:55.909230 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32d2cde-d29f-496e-a911-9e0566f20288 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:55.945239 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b373c6e0-ab7f-490b-a23c-93361fdd18fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:55.945905 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b373c6e0-ab7f-490b-a23c-93361fdd18fd None admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:55.946192 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b373c6e0-ab7f-490b-a23c-93361fdd18fd None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:55.946519 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b373c6e0-ab7f-490b-a23c-93361fdd18fd None admin] PATH_INFO: `/v3/endpoints` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:55.947251 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b373c6e0-ab7f-490b-a23c-93361fdd18fd None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:55.948687 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b373c6e0-ab7f-490b-a23c-93361fdd18fd None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:56.031009 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 510/1017] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Jul 12 17:20:55 2022] POST /identity/v3/endpoints => generated 334 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:20:56.436777 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f133b0f-9791-40b3-baf4-c8cbc82f1eaa None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:56.437305 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f133b0f-9791-40b3-baf4-c8cbc82f1eaa None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:56.437305 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f133b0f-9791-40b3-baf4-c8cbc82f1eaa None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:56.438824 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 508/1018] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:56.444077 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb29a3e-9500-4eb3-9449-c99a3f9503dc None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:56.444481 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb29a3e-9500-4eb3-9449-c99a3f9503dc None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:56.444647 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb29a3e-9500-4eb3-9449-c99a3f9503dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:56.461258 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-8fb29a3e-9500-4eb3-9449-c99a3f9503dc None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:56.462477 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-8fb29a3e-9500-4eb3-9449-c99a3f9503dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:56.465294 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:56.540276 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 511/1019] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:56 2022] POST /identity/v3/auth/tokens => generated 1863 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:20:56.566475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f133b0f-9791-40b3-baf4-c8cbc82f1eaa None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:56.599459 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:56.678984 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f133b0f-9791-40b3-baf4-c8cbc82f1eaa None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:56.714905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea70e74d-d4cb-416c-9a83-8b7a5a905cfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:56.715574 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea70e74d-d4cb-416c-9a83-8b7a5a905cfc None admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:56.715867 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea70e74d-d4cb-416c-9a83-8b7a5a905cfc None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:56.716149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea70e74d-d4cb-416c-9a83-8b7a5a905cfc None admin] PATH_INFO: `/v3/services` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:56.716878 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea70e74d-d4cb-416c-9a83-8b7a5a905cfc None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:56.718247 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea70e74d-d4cb-416c-9a83-8b7a5a905cfc None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:56.751490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aafd367-96ff-4a85-9440-c05bcc0d16a3 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:56.751784 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aafd367-96ff-4a85-9440-c05bcc0d16a3 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:56.752070 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aafd367-96ff-4a85-9440-c05bcc0d16a3 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:56.753475 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 512/1020] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 12 17:20:56.758600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69b7a0e-a237-422f-b9f8-aa7b84202610 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:56.758893 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69b7a0e-a237-422f-b9f8-aa7b84202610 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:56.759174 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69b7a0e-a237-422f-b9f8-aa7b84202610 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:56.777260 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-d69b7a0e-a237-422f-b9f8-aa7b84202610 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:56.778790 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-d69b7a0e-a237-422f-b9f8-aa7b84202610 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:56.808749 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 509/1021] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Jul 12 17:20:56 2022] POST /identity/v3/services => generated 242 bytes in 245 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:20:56.868779 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 513/1022] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:56 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 12 17:20:56.893952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea70e74d-d4cb-416c-9a83-8b7a5a905cfc None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:57.002287 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:57.014991 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea70e74d-d4cb-416c-9a83-8b7a5a905cfc None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:57.053182 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8b1a3a-ef22-4e27-93b7-cdcfae4e14d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:57.054337 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8b1a3a-ef22-4e27-93b7-cdcfae4e14d1 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:57.054797 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8b1a3a-ef22-4e27-93b7-cdcfae4e14d1 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:57.055233 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8b1a3a-ef22-4e27-93b7-cdcfae4e14d1 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:57.056581 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8b1a3a-ef22-4e27-93b7-cdcfae4e14d1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:57.058518 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8b1a3a-ef22-4e27-93b7-cdcfae4e14d1 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:57.060295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 510/1023] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:56 2022] GET /identity/v3/domains/default => generated 199 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:57.066130 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69b7a0e-a237-422f-b9f8-aa7b84202610 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:57.131452 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:57.173533 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69b7a0e-a237-422f-b9f8-aa7b84202610 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:57.207982 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4905c0-a35e-4cb9-85c6-2ad86e28fd42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:57.208668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4905c0-a35e-4cb9-85c6-2ad86e28fd42 None admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:57.208958 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4905c0-a35e-4cb9-85c6-2ad86e28fd42 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:57.209237 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4905c0-a35e-4cb9-85c6-2ad86e28fd42 None admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:57.209971 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f4905c0-a35e-4cb9-85c6-2ad86e28fd42 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:57.211436 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f4905c0-a35e-4cb9-85c6-2ad86e28fd42 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:57.353815 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 514/1024] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Jul 12 17:20:57 2022] POST /identity/v3/users => generated 285 bytes in 288 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:20:58.192601 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe829e65-2817-489e-80c6-1a264df2b4a9 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:58.193150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe829e65-2817-489e-80c6-1a264df2b4a9 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:58.193150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe829e65-2817-489e-80c6-1a264df2b4a9 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:58.194711 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 511/1025] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:58.200375 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b5bd1a-a8a0-4772-a512-2fc46005b1ce None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:58.200818 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b5bd1a-a8a0-4772-a512-2fc46005b1ce None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:58.200966 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b5bd1a-a8a0-4772-a512-2fc46005b1ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:58.218417 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-89b5bd1a-a8a0-4772-a512-2fc46005b1ce None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:58.219621 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-89b5bd1a-a8a0-4772-a512-2fc46005b1ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:58.287821 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 515/1026] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:58 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:20:58.316596 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe829e65-2817-489e-80c6-1a264df2b4a9 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:58.427711 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe829e65-2817-489e-80c6-1a264df2b4a9 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:58.463030 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c388631c-efff-4840-bfce-15d836d1e2fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:58.464496 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c388631c-efff-4840-bfce-15d836d1e2fa None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:58.464823 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c388631c-efff-4840-bfce-15d836d1e2fa None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:58.465068 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c388631c-efff-4840-bfce-15d836d1e2fa None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:58.475706 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c388631c-efff-4840-bfce-15d836d1e2fa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:20:58.484854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-c388631c-efff-4840-bfce-15d836d1e2fa None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 12 17:20:58.484854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:58.484854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:58.484854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:58.484854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:58.484854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:58.484854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:58.484854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:58.484854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:58.484854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:58.484854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:58.484854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:58.484854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:20:58.484854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:20:58.484854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:20:58.484854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:20:58.485905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:20:58.485905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:20:58.485905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:20:58.485905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:20:58.485905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:58.485905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:58.485905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:58.485905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:58.485905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:58.485905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:58.485905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:58.485905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:58.485905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:58.485905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:58.485905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:58.485905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:58.486816 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:58.486816 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:58.486816 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:58.486816 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:58.486816 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:20:58.486816 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:20:58.486816 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:20:58.486816 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:20:58.486816 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:20:58.486816 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:20:58.486816 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 12 17:20:58.486816 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:58.486816 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 512/1027] 10.222.0.16 () {56 vars in 1120 bytes} [Tue Jul 12 17:20:58 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:20:58.492783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b5bd1a-a8a0-4772-a512-2fc46005b1ce None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:58.544277 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:58.602343 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b5bd1a-a8a0-4772-a512-2fc46005b1ce None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:58.641226 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc263d8-ecfc-4603-a10a-8ad849342bb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:58.641907 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc263d8-ecfc-4603-a10a-8ad849342bb2 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:58.642197 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc263d8-ecfc-4603-a10a-8ad849342bb2 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:58.642503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc263d8-ecfc-4603-a10a-8ad849342bb2 None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:58.643295 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc263d8-ecfc-4603-a10a-8ad849342bb2 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:58.643605 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc263d8-ecfc-4603-a10a-8ad849342bb2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:58.644979 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc263d8-ecfc-4603-a10a-8ad849342bb2 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:58.658724 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 516/1028] 10.222.0.16 () {56 vars in 1115 bytes} [Tue Jul 12 17:20:58 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:58.660334 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c388631c-efff-4840-bfce-15d836d1e2fa None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:58.680207 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:58.765752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c388631c-efff-4840-bfce-15d836d1e2fa None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:58.802079 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275f08df-b1b2-4e05-b6c8-ab2be95b956e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:58.802966 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275f08df-b1b2-4e05-b6c8-ab2be95b956e None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:58.803261 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275f08df-b1b2-4e05-b6c8-ab2be95b956e None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:58.803539 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275f08df-b1b2-4e05-b6c8-ab2be95b956e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:58.814890 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275f08df-b1b2-4e05-b6c8-ab2be95b956e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:58.816346 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275f08df-b1b2-4e05-b6c8-ab2be95b956e None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:58.834992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-275f08df-b1b2-4e05-b6c8-ab2be95b956e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:58.834992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:58.834992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:58.834992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:58.834992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:58.834992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:58.834992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:58.834992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:58.834992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:58.834992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:58.834992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:58.834992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:58.834992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:58.834992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:58.834992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:58.835963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:58.835963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:58.835963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:58.835963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:58.835963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:58.835963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:58.835963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:58.835963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:58.835963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:58.835963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:58.835963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:58.835963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:58.835963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:58.835963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:58.835963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:58.835963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:58.835963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:58.841692 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:58.841692 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:58.841692 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:58.841692 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:58.841692 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 513/1029] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:20:58.843633 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc263d8-ecfc-4603-a10a-8ad849342bb2 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:58.911936 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5af9ab-14b6-440c-a127-819d69c50371 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:58.912266 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5af9ab-14b6-440c-a127-819d69c50371 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:58.912514 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5af9ab-14b6-440c-a127-819d69c50371 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:58.914110 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 514/1030] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 12 17:20:58.919496 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ed24a8-b4aa-4f64-982a-b3809a4cc648 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:58.919790 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ed24a8-b4aa-4f64-982a-b3809a4cc648 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:58.920080 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ed24a8-b4aa-4f64-982a-b3809a4cc648 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:58.937135 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-a6ed24a8-b4aa-4f64-982a-b3809a4cc648 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:58.938315 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-a6ed24a8-b4aa-4f64-982a-b3809a4cc648 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:58.954017 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc263d8-ecfc-4603-a10a-8ad849342bb2 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:58.963928 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:58.990364 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14aef1e3-191a-4d83-a7b5-7fdeda3b7e90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:58.991364 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14aef1e3-191a-4d83-a7b5-7fdeda3b7e90 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:58.991656 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14aef1e3-191a-4d83-a7b5-7fdeda3b7e90 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:58.991936 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14aef1e3-191a-4d83-a7b5-7fdeda3b7e90 None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:58.992691 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14aef1e3-191a-4d83-a7b5-7fdeda3b7e90 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:58.993005 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14aef1e3-191a-4d83-a7b5-7fdeda3b7e90 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:58.995693 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14aef1e3-191a-4d83-a7b5-7fdeda3b7e90 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:59.003517 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:59.010110 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 515/1031] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:58 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 12 17:20:59.010769 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 517/1032] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:59.018474 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ed24a8-b4aa-4f64-982a-b3809a4cc648 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:59.032398 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14aef1e3-191a-4d83-a7b5-7fdeda3b7e90 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:59.134289 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14aef1e3-191a-4d83-a7b5-7fdeda3b7e90 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:59.134856 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ed24a8-b4aa-4f64-982a-b3809a4cc648 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:59.169393 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10f78a9-14b1-4551-9e2a-420a739d0f6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:59.170189 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10f78a9-14b1-4551-9e2a-420a739d0f6b None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:59.170489 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10f78a9-14b1-4551-9e2a-420a739d0f6b None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:59.170774 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10f78a9-14b1-4551-9e2a-420a739d0f6b None admin] PATH_INFO: `/v3/services/image` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:59.171438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10f78a9-14b1-4551-9e2a-420a739d0f6b None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:59.172794 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10f78a9-14b1-4551-9e2a-420a739d0f6b None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:59.175408 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9056771-857b-418f-87e8-c3a874848330 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:59.176208 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9056771-857b-418f-87e8-c3a874848330 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:59.176520 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9056771-857b-418f-87e8-c3a874848330 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:59.176803 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9056771-857b-418f-87e8-c3a874848330 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:59.185312 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9056771-857b-418f-87e8-c3a874848330 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:59.185575 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-a10f78a9-14b1-4551-9e2a-420a739d0f6b None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Jul 12 17:20:59.185575 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:59.185575 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 12 17:20:59.185575 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 12 17:20:59.185575 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 12 17:20:59.185575 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 12 17:20:59.185575 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 12 17:20:59.185575 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:59.185575 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 12 17:20:59.185575 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:59.185575 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 12 17:20:59.185575 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:59.185575 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:59.185575 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:59.185575 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:59.185575 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:59.185575 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:59.190555 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:59.190555 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:59.190555 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:59.190555 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:59.190555 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:59.190555 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:59.190555 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 12 17:20:59.190555 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 12 17:20:59.190555 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 12 17:20:59.190555 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 12 17:20:59.190555 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:59.190555 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:59.190555 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:59.190555 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:59.190555 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:59.190555 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:59.191473 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:59.191473 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:59.191473 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:59.191473 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:59.191473 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:59.191473 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:59.191473 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:59.191473 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:59.191473 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:59.191473 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:59.191473 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 12 17:20:59.191473 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:20:59.191473 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 12 17:20:59.191473 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:59.192234 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9056771-857b-418f-87e8-c3a874848330 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:59.192310 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 518/1033] 10.222.0.16 () {56 vars in 1105 bytes} [Tue Jul 12 17:20:59 2022] GET /identity/v3/services/image => generated 86 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:20:59.193613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10f78a9-14b1-4551-9e2a-420a739d0f6b None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:59.204354 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-f9056771-857b-418f-87e8-c3a874848330 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 12 17:20:59.204354 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:59.204354 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:59.204354 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:59.204354 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:59.204354 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:59.204354 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:59.204354 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:59.204354 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:59.204354 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:59.204354 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:59.204354 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:59.204354 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:20:59.204354 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:20:59.204354 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:20:59.204354 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:20:59.205676 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:59.205676 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:59.205676 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:20:59.205676 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:59.205676 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:20:59.205676 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:20:59.205676 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:59.205676 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:59.205676 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:59.205676 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:59.205676 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:59.205676 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:59.205676 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:59.205676 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:59.205676 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:59.205676 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:59.205676 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:59.205676 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:59.206686 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:59.206686 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:59.206686 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:20:59.206686 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:20:59.206686 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:20:59.206686 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:20:59.206686 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:20:59.206686 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:20:59.206686 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 12 17:20:59.206686 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:20:59.206686 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 516/1034] 10.222.0.16 () {56 vars in 1093 bytes} [Tue Jul 12 17:20:59 2022] GET /identity/v3/users/nova => generated 82 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:20:59.211213 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9056771-857b-418f-87e8-c3a874848330 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:59.234370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:59.276434 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:59.306975 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10f78a9-14b1-4551-9e2a-420a739d0f6b None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:59.321507 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9056771-857b-418f-87e8-c3a874848330 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:59.345966 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7cb853-4b76-4a0a-a4d6-0cb972fb8d0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:59.346933 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7cb853-4b76-4a0a-a4d6-0cb972fb8d0c None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:59.347402 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7cb853-4b76-4a0a-a4d6-0cb972fb8d0c None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:59.347687 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7cb853-4b76-4a0a-a4d6-0cb972fb8d0c None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:59.348400 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7cb853-4b76-4a0a-a4d6-0cb972fb8d0c None admin] RBAC: Adding query filter params (name=image) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:59.348714 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7cb853-4b76-4a0a-a4d6-0cb972fb8d0c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:59.350334 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7cb853-4b76-4a0a-a4d6-0cb972fb8d0c None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:59.358478 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 519/1035] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:59 2022] GET /identity/v3/services?name=image => generated 122 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:20:59.361781 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7cb853-4b76-4a0a-a4d6-0cb972fb8d0c None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:59.364023 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3eaa806-cc8f-4db6-b54e-8247466cd664 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:59.364912 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3eaa806-cc8f-4db6-b54e-8247466cd664 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:59.366074 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3eaa806-cc8f-4db6-b54e-8247466cd664 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:59.366375 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3eaa806-cc8f-4db6-b54e-8247466cd664 None admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:59.367257 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3eaa806-cc8f-4db6-b54e-8247466cd664 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:59.367573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3eaa806-cc8f-4db6-b54e-8247466cd664 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:59.369014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3eaa806-cc8f-4db6-b54e-8247466cd664 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:59.393442 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 517/1036] 10.222.0.16 () {56 vars in 1133 bytes} [Tue Jul 12 17:20:59 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:20:59.394592 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3eaa806-cc8f-4db6-b54e-8247466cd664 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:59.465023 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:59.471088 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7cb853-4b76-4a0a-a4d6-0cb972fb8d0c None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:59.505282 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc4d70f-42cf-4976-9651-6a9a1c1b1059 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:59.505616 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3eaa806-cc8f-4db6-b54e-8247466cd664 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:59.505949 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc4d70f-42cf-4976-9651-6a9a1c1b1059 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:59.506237 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc4d70f-42cf-4976-9651-6a9a1c1b1059 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:59.506552 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc4d70f-42cf-4976-9651-6a9a1c1b1059 None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:59.507268 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc4d70f-42cf-4976-9651-6a9a1c1b1059 None admin] RBAC: Adding query filter params (type=image) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:59.507580 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc4d70f-42cf-4976-9651-6a9a1c1b1059 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:59.508961 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc4d70f-42cf-4976-9651-6a9a1c1b1059 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:59.520144 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 520/1037] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:59 2022] GET /identity/v3/services?type=image => generated 350 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:20:59.520144 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea0448e-f96c-4316-8959-2287c12d42ab None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:59.520144 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea0448e-f96c-4316-8959-2287c12d42ab None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:59.520144 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea0448e-f96c-4316-8959-2287c12d42ab None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:59.520144 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 521/1038] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:20:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:20:59.521068 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea0448e-f96c-4316-8959-2287c12d42ab None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:59.526753 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:59.545427 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ecec14e-9da3-4524-b7d6-56e20a6ce2cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:59.546238 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecec14e-9da3-4524-b7d6-56e20a6ce2cb None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:59.546566 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecec14e-9da3-4524-b7d6-56e20a6ce2cb None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:59.546851 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecec14e-9da3-4524-b7d6-56e20a6ce2cb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:59.615957 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ecec14e-9da3-4524-b7d6-56e20a6ce2cb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:59.617424 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ecec14e-9da3-4524-b7d6-56e20a6ce2cb None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:59.631457 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea0448e-f96c-4316-8959-2287c12d42ab None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:59.666310 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de9e2386-f5f9-4103-8c5e-d3a8292cfddd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:59.667020 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de9e2386-f5f9-4103-8c5e-d3a8292cfddd None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:59.667305 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de9e2386-f5f9-4103-8c5e-d3a8292cfddd None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:59.667583 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de9e2386-f5f9-4103-8c5e-d3a8292cfddd None admin] PATH_INFO: `/v3/endpoints` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:59.668371 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de9e2386-f5f9-4103-8c5e-d3a8292cfddd None admin] RBAC: Adding query filter params (interface=public, service_id=16688884e6a245e69e76381a3cb39ad2, region_id=RegionOne) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:20:59.668686 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de9e2386-f5f9-4103-8c5e-d3a8292cfddd None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:59.670053 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de9e2386-f5f9-4103-8c5e-d3a8292cfddd None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:59.675230 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 522/1039] 10.222.0.16 () {56 vars in 1251 bytes} [Tue Jul 12 17:20:59 2022] GET /identity/v3/endpoints?service_id=16688884e6a245e69e76381a3cb39ad2&interface=public®ion_id=RegionOne => generated 194 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:20:59.676402 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377ab354-2135-4e21-ad25-25234059360b None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:59.676696 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377ab354-2135-4e21-ad25-25234059360b None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:59.676976 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377ab354-2135-4e21-ad25-25234059360b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:59.692926 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-377ab354-2135-4e21-ad25-25234059360b None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:20:59.694115 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-377ab354-2135-4e21-ad25-25234059360b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:20:59.711646 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:59.724587 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 518/1040] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:20:59 2022] GET /identity/v3/auth/tokens => generated 1959 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 12 17:20:59.725428 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ecec14e-9da3-4524-b7d6-56e20a6ce2cb None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:59.742463 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:59.767495 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 523/1041] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:20:59 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:20:59.768573 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377ab354-2135-4e21-ad25-25234059360b None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:59.826349 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ecec14e-9da3-4524-b7d6-56e20a6ce2cb None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:59.860835 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5608f0a-fd2a-4c6e-895c-9f7f6a367d82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:59.861517 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5608f0a-fd2a-4c6e-895c-9f7f6a367d82 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:59.861803 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5608f0a-fd2a-4c6e-895c-9f7f6a367d82 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:59.862082 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5608f0a-fd2a-4c6e-895c-9f7f6a367d82 None admin] PATH_INFO: `/v3/services` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:59.862816 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5608f0a-fd2a-4c6e-895c-9f7f6a367d82 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:59.864164 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5608f0a-fd2a-4c6e-895c-9f7f6a367d82 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:59.868319 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377ab354-2135-4e21-ad25-25234059360b None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:20:59.871467 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 519/1042] 10.222.0.16 () {56 vars in 1087 bytes} [Tue Jul 12 17:20:59 2022] GET /identity/v3/services => generated 1957 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:20:59.871849 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5608f0a-fd2a-4c6e-895c-9f7f6a367d82 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:59.909681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30e3a42-67a6-45bd-9471-d200f4aa2e13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:20:59.910532 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30e3a42-67a6-45bd-9471-d200f4aa2e13 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:20:59.910823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30e3a42-67a6-45bd-9471-d200f4aa2e13 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:20:59.911107 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30e3a42-67a6-45bd-9471-d200f4aa2e13 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:20:59.922194 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30e3a42-67a6-45bd-9471-d200f4aa2e13 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:20:59.923726 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30e3a42-67a6-45bd-9471-d200f4aa2e13 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:20:59.943157 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-d30e3a42-67a6-45bd-9471-d200f4aa2e13 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:59.943157 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:20:59.943157 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:20:59.943157 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:20:59.943157 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:20:59.943157 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:20:59.943157 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:20:59.943157 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:20:59.943157 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:20:59.943157 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:20:59.943157 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:20:59.943157 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:20:59.943157 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:20:59.943157 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:20:59.943157 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:20:59.950916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:20:59.950916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:20:59.950916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:20:59.950916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:20:59.950916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:20:59.950916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:20:59.950916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:20:59.950916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:20:59.950916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:20:59.950916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:20:59.950916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:20:59.950916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:20:59.950916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:20:59.950916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:20:59.950916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:20:59.950916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:20:59.950916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:20:59.952035 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:20:59.952035 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:20:59.952035 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:20:59.952035 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:20:59.952035 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 524/1043] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:20:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:20:59.953670 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30e3a42-67a6-45bd-9471-d200f4aa2e13 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:20:59.962727 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:59.973285 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:20:59.997403 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5608f0a-fd2a-4c6e-895c-9f7f6a367d82 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:00.037336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db356d67-e548-4206-af0c-74174eee8f55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:00.038143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db356d67-e548-4206-af0c-74174eee8f55 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:00.038455 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db356d67-e548-4206-af0c-74174eee8f55 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:00.038749 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db356d67-e548-4206-af0c-74174eee8f55 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:00.049686 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db356d67-e548-4206-af0c-74174eee8f55 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:21:00.058268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-db356d67-e548-4206-af0c-74174eee8f55 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:21:00.058268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:21:00.058268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:21:00.058268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:21:00.058268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:21:00.058268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:21:00.058268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:21:00.058268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:21:00.058268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:21:00.058268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:21:00.058268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:21:00.058268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:21:00.058268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:21:00.058268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:21:00.058268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:21:00.058268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:21:00.059683 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:21:00.059683 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:21:00.059683 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:21:00.059683 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:21:00.059683 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:21:00.059683 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:21:00.059683 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:21:00.059683 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:21:00.059683 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:21:00.059683 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:21:00.059683 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:21:00.059683 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:21:00.059683 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:21:00.059683 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:21:00.059683 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:21:00.059683 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:21:00.060751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:21:00.060751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:21:00.060751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:21:00.060751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:21:00.060751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:21:00.060751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:21:00.060751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:21:00.060751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:21:00.060751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:21:00.060751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:21:00.060751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:21:00.060751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:21:00.060751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 520/1044] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Jul 12 17:20:59 2022] GET /identity/v3/roles/admin => generated 83 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:21:00.066580 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db356d67-e548-4206-af0c-74174eee8f55 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:00.070078 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30e3a42-67a6-45bd-9471-d200f4aa2e13 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:00.109995 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394a678c-0f4d-4f37-bcb2-ebb0e09a6207 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:00.110620 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394a678c-0f4d-4f37-bcb2-ebb0e09a6207 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:00.110914 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394a678c-0f4d-4f37-bcb2-ebb0e09a6207 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:00.111198 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394a678c-0f4d-4f37-bcb2-ebb0e09a6207 None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:00.111935 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-394a678c-0f4d-4f37-bcb2-ebb0e09a6207 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:21:00.112252 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-394a678c-0f4d-4f37-bcb2-ebb0e09a6207 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:00.113633 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-394a678c-0f4d-4f37-bcb2-ebb0e09a6207 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:00.126228 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 525/1045] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:20:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:21:00.131813 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394a678c-0f4d-4f37-bcb2-ebb0e09a6207 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:00.179072 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db356d67-e548-4206-af0c-74174eee8f55 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:00.204857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:00.215461 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff8188b-130a-4317-9cfd-2384859fc312 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:00.216140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff8188b-130a-4317-9cfd-2384859fc312 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:00.216433 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff8188b-130a-4317-9cfd-2384859fc312 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:00.216717 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff8188b-130a-4317-9cfd-2384859fc312 None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:00.217531 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff8188b-130a-4317-9cfd-2384859fc312 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:21:00.217845 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff8188b-130a-4317-9cfd-2384859fc312 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:00.219261 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff8188b-130a-4317-9cfd-2384859fc312 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:00.228730 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 521/1046] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:21:00 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:00.235160 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff8188b-130a-4317-9cfd-2384859fc312 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:00.238041 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:00.246385 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394a678c-0f4d-4f37-bcb2-ebb0e09a6207 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:00.282981 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33802618-4053-4c13-a3a4-005db4926129 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:00.283872 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33802618-4053-4c13-a3a4-005db4926129 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:00.284205 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33802618-4053-4c13-a3a4-005db4926129 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:00.284464 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33802618-4053-4c13-a3a4-005db4926129 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:00.291365 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33802618-4053-4c13-a3a4-005db4926129 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:00.293848 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33802618-4053-4c13-a3a4-005db4926129 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:00.311418 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-33802618-4053-4c13-a3a4-005db4926129 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:21:00.311418 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:21:00.311418 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:21:00.311418 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:21:00.311418 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:21:00.311418 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:21:00.311418 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:21:00.311418 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:21:00.311418 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:21:00.311418 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:21:00.311418 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:21:00.311418 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:21:00.311418 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:21:00.311418 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:21:00.311418 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:21:00.312540 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:21:00.312540 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:21:00.312540 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:21:00.312540 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:21:00.312540 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:21:00.312540 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:21:00.312540 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:21:00.312540 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:21:00.312540 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:21:00.312540 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:21:00.312540 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:21:00.312540 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:21:00.312540 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:21:00.312540 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:21:00.312540 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:21:00.312540 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:21:00.312540 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:21:00.313879 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:21:00.313879 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:21:00.313879 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:21:00.313879 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:21:00.313879 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:21:00.313879 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:21:00.313879 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:21:00.313879 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:21:00.315761 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 526/1047] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:21:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:21:00.321447 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33802618-4053-4c13-a3a4-005db4926129 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:00.348854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff8188b-130a-4317-9cfd-2384859fc312 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:00.391890 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9adbf718-0f7c-45be-bb62-25527d000b4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:00.392726 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9adbf718-0f7c-45be-bb62-25527d000b4b None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:00.393063 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9adbf718-0f7c-45be-bb62-25527d000b4b None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:00.393355 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9adbf718-0f7c-45be-bb62-25527d000b4b None admin] PATH_INFO: `/v3/users/8754002d8c5a4870b0ecad29b09b107f` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:00.407680 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9adbf718-0f7c-45be-bb62-25527d000b4b None admin] RBAC: Authorizing `identity:get_user(user_id=8754002d8c5a4870b0ecad29b09b107f)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:00.409147 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9adbf718-0f7c-45be-bb62-25527d000b4b None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:00.415797 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 522/1048] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:21:00 2022] GET /identity/v3/users/8754002d8c5a4870b0ecad29b09b107f => generated 285 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:21:00.416712 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9adbf718-0f7c-45be-bb62-25527d000b4b None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:00.431589 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33802618-4053-4c13-a3a4-005db4926129 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:00.459484 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:00.469736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3333d13-3645-45e2-a2ca-fe59683f57ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:00.470375 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3333d13-3645-45e2-a2ca-fe59683f57ed None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:00.478568 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3333d13-3645-45e2-a2ca-fe59683f57ed None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:00.478568 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3333d13-3645-45e2-a2ca-fe59683f57ed None admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:00.478568 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3333d13-3645-45e2-a2ca-fe59683f57ed None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:21:00.478568 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3333d13-3645-45e2-a2ca-fe59683f57ed None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:00.478568 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3333d13-3645-45e2-a2ca-fe59683f57ed None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:00.479171 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:00.487116 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 527/1049] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Jul 12 17:21:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:21:00.492702 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3333d13-3645-45e2-a2ca-fe59683f57ed None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:00.522228 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9adbf718-0f7c-45be-bb62-25527d000b4b None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:00.561623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900256e3-1c25-42d9-bc49-fbb3ea138280 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:00.562286 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900256e3-1c25-42d9-bc49-fbb3ea138280 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:00.562733 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900256e3-1c25-42d9-bc49-fbb3ea138280 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:00.562914 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900256e3-1c25-42d9-bc49-fbb3ea138280 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:00.608678 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3333d13-3645-45e2-a2ca-fe59683f57ed None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:00.647928 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4e757f-f48f-413c-9b8b-7967cfb2f7b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:00.648743 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4e757f-f48f-413c-9b8b-7967cfb2f7b9 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:00.649015 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4e757f-f48f-413c-9b8b-7967cfb2f7b9 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:00.649346 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4e757f-f48f-413c-9b8b-7967cfb2f7b9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:00.650196 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4e757f-f48f-413c-9b8b-7967cfb2f7b9 None admin] RBAC: Adding query filter params (role.id=feb6a96f7b6c4f0a8d64be226f85ee6c, scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2, user.id=9012d5b88f554dc596582a7693c2a72a) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:21:00.650550 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4e757f-f48f-413c-9b8b-7967cfb2f7b9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:00.650687 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-900256e3-1c25-42d9-bc49-fbb3ea138280 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:00.652610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-900256e3-1c25-42d9-bc49-fbb3ea138280 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:00.652978 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4e757f-f48f-413c-9b8b-7967cfb2f7b9 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:00.670523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 528/1050] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:21:00 2022] GET /identity/v3/role_assignments?user.id=9012d5b88f554dc596582a7693c2a72a&scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2&role.id=feb6a96f7b6c4f0a8d64be226f85ee6c => generated 259 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:21:00.730935 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:00.777881 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 523/1051] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:21:00 2022] GET /identity/v3/auth/tokens => generated 1959 bytes in 361 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 12 17:21:00.783775 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4e757f-f48f-413c-9b8b-7967cfb2f7b9 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:00.836124 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:00.888276 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4e757f-f48f-413c-9b8b-7967cfb2f7b9 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:00.925390 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896c4987-c112-4094-a28a-af181c23f492 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:00.926169 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896c4987-c112-4094-a28a-af181c23f492 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:00.926471 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896c4987-c112-4094-a28a-af181c23f492 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:00.926752 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896c4987-c112-4094-a28a-af181c23f492 None admin] PATH_INFO: `/v3/projects/962bcc233b6f45b4a6adc14a7efd4ccf` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:00.927922 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896c4987-c112-4094-a28a-af181c23f492 None admin] RBAC: Authorizing `identity:get_project(project_id=962bcc233b6f45b4a6adc14a7efd4ccf)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:00.929411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896c4987-c112-4094-a28a-af181c23f492 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:00.931304 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 529/1052] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:21:00 2022] GET /identity/v3/projects/962bcc233b6f45b4a6adc14a7efd4ccf => generated 311 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:00.937867 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900256e3-1c25-42d9-bc49-fbb3ea138280 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:01.046966 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900256e3-1c25-42d9-bc49-fbb3ea138280 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:01.082355 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b06fb71-1240-462b-a1bb-9f35f6dbf346 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:01.083054 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b06fb71-1240-462b-a1bb-9f35f6dbf346 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:01.083344 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b06fb71-1240-462b-a1bb-9f35f6dbf346 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:01.083623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b06fb71-1240-462b-a1bb-9f35f6dbf346 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:01.084456 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b06fb71-1240-462b-a1bb-9f35f6dbf346 None admin] RBAC: Adding query filter params (role.id=eb05bf286c894ea9bf44a6c023a4323b, scope.project.id=962bcc233b6f45b4a6adc14a7efd4ccf, user.id=8754002d8c5a4870b0ecad29b09b107f) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:21:01.084779 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b06fb71-1240-462b-a1bb-9f35f6dbf346 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:01.086237 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b06fb71-1240-462b-a1bb-9f35f6dbf346 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:01.097104 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 524/1053] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:21:00 2022] GET /identity/v3/role_assignments?user.id=8754002d8c5a4870b0ecad29b09b107f&scope.project.id=962bcc233b6f45b4a6adc14a7efd4ccf&role.id=eb05bf286c894ea9bf44a6c023a4323b => generated 259 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:21:01.979357 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b011b2-2cc4-48cf-bd3c-d859b7097b93 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:01.979994 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b011b2-2cc4-48cf-bd3c-d859b7097b93 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:01.979994 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b011b2-2cc4-48cf-bd3c-d859b7097b93 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:01.981401 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 530/1054] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:21:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:21:01.987647 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a96921e-5239-4746-b605-0849fac90823 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:01.988054 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a96921e-5239-4746-b605-0849fac90823 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:01.988290 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a96921e-5239-4746-b605-0849fac90823 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:02.006812 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-1a96921e-5239-4746-b605-0849fac90823 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:21:02.008014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-1a96921e-5239-4746-b605-0849fac90823 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:21:02.026852 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:02.077638 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 525/1055] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:21:01 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:21:02.100684 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b011b2-2cc4-48cf-bd3c-d859b7097b93 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:02.208507 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b011b2-2cc4-48cf-bd3c-d859b7097b93 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:02.229244 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:02.247110 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce243e9d-95d3-4b71-9ffb-8e78b4ce6435 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:02.247840 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce243e9d-95d3-4b71-9ffb-8e78b4ce6435 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:02.248130 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce243e9d-95d3-4b71-9ffb-8e78b4ce6435 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:02.248411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce243e9d-95d3-4b71-9ffb-8e78b4ce6435 None admin] PATH_INFO: `/v3/services/image` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:02.249073 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce243e9d-95d3-4b71-9ffb-8e78b4ce6435 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:02.250512 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce243e9d-95d3-4b71-9ffb-8e78b4ce6435 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:02.262533 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-ce243e9d-95d3-4b71-9ffb-8e78b4ce6435 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Jul 12 17:21:02.262533 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:21:02.262533 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 12 17:21:02.262533 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 12 17:21:02.262533 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 12 17:21:02.262533 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 12 17:21:02.262533 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 12 17:21:02.262533 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:21:02.262533 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 12 17:21:02.262533 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:21:02.262533 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 12 17:21:02.262533 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:21:02.262533 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:21:02.262533 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:21:02.262533 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:21:02.262533 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:21:02.262533 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:21:02.263767 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:21:02.263767 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:21:02.263767 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:21:02.263767 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:21:02.263767 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:21:02.263767 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:21:02.263767 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 12 17:21:02.263767 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 12 17:21:02.263767 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 12 17:21:02.263767 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 12 17:21:02.263767 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:21:02.263767 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:21:02.263767 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:21:02.263767 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:21:02.263767 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:21:02.263767 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:21:02.264692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:21:02.264692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:21:02.264692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:21:02.264692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:21:02.264692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:21:02.264692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:21:02.264692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:21:02.264692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:21:02.264692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:21:02.264692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:21:02.264692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 12 17:21:02.264692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:21:02.264692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 12 17:21:02.264692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:21:02.264692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 531/1056] 10.222.0.16 () {56 vars in 1105 bytes} [Tue Jul 12 17:21:02 2022] GET /identity/v3/services/image => generated 86 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:21:02.271806 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a96921e-5239-4746-b605-0849fac90823 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:02.377713 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a96921e-5239-4746-b605-0849fac90823 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:02.413292 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf9e284-9e3c-4d17-90fc-efec39f4b35e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:02.413960 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf9e284-9e3c-4d17-90fc-efec39f4b35e None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:02.414249 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf9e284-9e3c-4d17-90fc-efec39f4b35e None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:02.414565 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf9e284-9e3c-4d17-90fc-efec39f4b35e None admin] PATH_INFO: `/v3/services` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:02.415280 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf9e284-9e3c-4d17-90fc-efec39f4b35e None admin] RBAC: Adding query filter params (name=image) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:21:02.415614 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf9e284-9e3c-4d17-90fc-efec39f4b35e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:02.417571 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf9e284-9e3c-4d17-90fc-efec39f4b35e None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:02.427283 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 526/1057] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:21:02 2022] GET /identity/v3/services?name=image => generated 122 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:02.428431 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce243e9d-95d3-4b71-9ffb-8e78b4ce6435 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:02.532466 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce243e9d-95d3-4b71-9ffb-8e78b4ce6435 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:02.572022 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198a2fdb-0163-408e-b969-11fb3c8cffdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:02.574608 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198a2fdb-0163-408e-b969-11fb3c8cffdc None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:02.574608 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198a2fdb-0163-408e-b969-11fb3c8cffdc None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:02.574608 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198a2fdb-0163-408e-b969-11fb3c8cffdc None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:02.574608 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-198a2fdb-0163-408e-b969-11fb3c8cffdc None admin] RBAC: Adding query filter params (type=image) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:21:02.574938 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-198a2fdb-0163-408e-b969-11fb3c8cffdc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:02.576468 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-198a2fdb-0163-408e-b969-11fb3c8cffdc None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:02.583381 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 532/1058] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:21:02 2022] GET /identity/v3/services?type=image => generated 350 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:21:02.589692 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf9e284-9e3c-4d17-90fc-efec39f4b35e None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:02.609547 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:02.696777 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf9e284-9e3c-4d17-90fc-efec39f4b35e None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:02.718765 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2b2f07-b1b5-4dfc-ad43-795329ccf2c4 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:02.719062 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2b2f07-b1b5-4dfc-ad43-795329ccf2c4 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:02.719340 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2b2f07-b1b5-4dfc-ad43-795329ccf2c4 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:02.720710 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 533/1059] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:21:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:21:02.725929 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00b9187-199c-4d44-a835-1f5e107fee8d None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:02.726222 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00b9187-199c-4d44-a835-1f5e107fee8d None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:02.726530 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00b9187-199c-4d44-a835-1f5e107fee8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:02.734261 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d475d6-7ae2-48f5-879b-d105c396301c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:02.734961 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d475d6-7ae2-48f5-879b-d105c396301c None admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:02.735252 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d475d6-7ae2-48f5-879b-d105c396301c None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:02.735533 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d475d6-7ae2-48f5-879b-d105c396301c None admin] PATH_INFO: `/v3/endpoints` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:02.736292 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d475d6-7ae2-48f5-879b-d105c396301c None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:02.737686 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d475d6-7ae2-48f5-879b-d105c396301c None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:02.743278 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-e00b9187-199c-4d44-a835-1f5e107fee8d None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:21:02.744607 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-e00b9187-199c-4d44-a835-1f5e107fee8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:21:02.788321 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:02.810184 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 527/1060] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Jul 12 17:21:02 2022] POST /identity/v3/endpoints => generated 327 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:21:02.830690 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 534/1061] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:21:02 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 12 17:21:02.855768 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d475d6-7ae2-48f5-879b-d105c396301c None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:02.946299 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:02.977578 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d475d6-7ae2-48f5-879b-d105c396301c None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:03.014105 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6241b2c3-7412-46b1-a18e-4cf891d5b026 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:03.014980 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6241b2c3-7412-46b1-a18e-4cf891d5b026 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:03.015272 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6241b2c3-7412-46b1-a18e-4cf891d5b026 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:03.015554 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6241b2c3-7412-46b1-a18e-4cf891d5b026 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:03.026917 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6241b2c3-7412-46b1-a18e-4cf891d5b026 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:21:03.035108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-6241b2c3-7412-46b1-a18e-4cf891d5b026 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 12 17:21:03.035108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:21:03.035108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:21:03.035108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:21:03.035108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:21:03.035108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:21:03.035108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:21:03.035108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:21:03.035108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:21:03.035108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:21:03.035108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:21:03.035108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:21:03.035108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:21:03.035108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:21:03.035108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:21:03.035108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:21:03.036065 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:21:03.036065 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:21:03.036065 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:21:03.036065 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:21:03.036065 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:21:03.036065 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:21:03.036065 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:21:03.036065 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:21:03.036065 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:21:03.036065 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:21:03.036065 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:21:03.036065 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:21:03.036065 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:21:03.036065 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:21:03.036065 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:21:03.036065 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:21:03.036931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:21:03.036931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:21:03.036931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:21:03.036931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:21:03.036931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:21:03.036931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:21:03.036931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:21:03.036931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:21:03.036931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:21:03.036931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:21:03.036931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 12 17:21:03.036931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:21:03.036931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 528/1062] 10.222.0.16 () {56 vars in 1120 bytes} [Tue Jul 12 17:21:02 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:21:03.044940 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e00b9187-199c-4d44-a835-1f5e107fee8d None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:03.147537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e00b9187-199c-4d44-a835-1f5e107fee8d None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:03.171952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd2a08a-2f51-4f60-bc12-54ea3a27587a None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:03.172251 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd2a08a-2f51-4f60-bc12-54ea3a27587a None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:03.172535 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd2a08a-2f51-4f60-bc12-54ea3a27587a None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:03.173925 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 529/1063] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:21:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:21:03.179308 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d33a8f1-4fed-4616-ae81-a595c640202f None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:03.179600 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d33a8f1-4fed-4616-ae81-a595c640202f None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:03.179884 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d33a8f1-4fed-4616-ae81-a595c640202f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:03.186041 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23f1803-b496-4933-abff-5c4c86cafa19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:03.186761 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23f1803-b496-4933-abff-5c4c86cafa19 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:03.187050 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23f1803-b496-4933-abff-5c4c86cafa19 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:03.187329 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23f1803-b496-4933-abff-5c4c86cafa19 None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:03.188155 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23f1803-b496-4933-abff-5c4c86cafa19 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:21:03.188779 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23f1803-b496-4933-abff-5c4c86cafa19 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:03.190208 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23f1803-b496-4933-abff-5c4c86cafa19 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:03.199108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-1d33a8f1-4fed-4616-ae81-a595c640202f None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:21:03.204986 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 535/1064] 10.222.0.16 () {56 vars in 1115 bytes} [Tue Jul 12 17:21:03 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:21:03.205137 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-1d33a8f1-4fed-4616-ae81-a595c640202f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:21:03.205787 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23f1803-b496-4933-abff-5c4c86cafa19 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:03.245060 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:03.274172 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 530/1065] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:21:03 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 12 17:21:03.297729 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d33a8f1-4fed-4616-ae81-a595c640202f None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:03.325152 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23f1803-b496-4933-abff-5c4c86cafa19 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:03.360403 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c6af82-ffd0-47a6-8648-69705a26cb36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:03.361202 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c6af82-ffd0-47a6-8648-69705a26cb36 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:03.361511 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c6af82-ffd0-47a6-8648-69705a26cb36 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:03.361772 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c6af82-ffd0-47a6-8648-69705a26cb36 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:03.365360 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:03.374321 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8c6af82-ffd0-47a6-8648-69705a26cb36 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:03.379716 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8c6af82-ffd0-47a6-8648-69705a26cb36 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:03.393681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-f8c6af82-ffd0-47a6-8648-69705a26cb36 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:21:03.393681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:21:03.393681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:21:03.393681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:21:03.393681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:21:03.393681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:21:03.393681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:21:03.393681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:21:03.393681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:21:03.393681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:21:03.393681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:21:03.393681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:21:03.393681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:21:03.393681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:21:03.393681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:21:03.394708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:21:03.394708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:21:03.394708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:21:03.394708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:21:03.394708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:21:03.394708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:21:03.394708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:21:03.394708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:21:03.394708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:21:03.394708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:21:03.394708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:21:03.394708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:21:03.394708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:21:03.394708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:21:03.394708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:21:03.394708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:21:03.394708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:21:03.395640 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:21:03.395640 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:21:03.395640 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:21:03.395640 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:21:03.395640 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 536/1066] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:21:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:21:03.400689 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c6af82-ffd0-47a6-8648-69705a26cb36 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:03.411356 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d33a8f1-4fed-4616-ae81-a595c640202f None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:03.448605 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58743014-aaa8-4d53-bcab-68041a59a70d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:03.449406 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58743014-aaa8-4d53-bcab-68041a59a70d None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:03.449692 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58743014-aaa8-4d53-bcab-68041a59a70d None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:03.449970 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58743014-aaa8-4d53-bcab-68041a59a70d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:03.460732 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58743014-aaa8-4d53-bcab-68041a59a70d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:21:03.468521 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-58743014-aaa8-4d53-bcab-68041a59a70d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:21:03.468521 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:21:03.468521 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:21:03.468521 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:21:03.468521 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:21:03.468521 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:21:03.468521 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:21:03.468521 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:21:03.468521 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:21:03.468521 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:21:03.468521 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:21:03.468521 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:21:03.468521 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:21:03.468521 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:21:03.468521 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:21:03.468521 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:21:03.469473 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:21:03.469473 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:21:03.469473 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:21:03.469473 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:21:03.469473 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:21:03.469473 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:21:03.469473 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:21:03.469473 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:21:03.469473 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:21:03.469473 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:21:03.469473 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:21:03.469473 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:21:03.469473 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:21:03.469473 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:21:03.469473 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:21:03.469473 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:21:03.470454 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:21:03.470454 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:21:03.470454 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:21:03.470454 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:21:03.470454 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:21:03.470454 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:21:03.470454 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:21:03.470454 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:21:03.470454 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:21:03.470454 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:21:03.470454 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:21:03.470454 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:21:03.470454 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 531/1067] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Jul 12 17:21:03 2022] GET /identity/v3/roles/admin => generated 83 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:21:03.475877 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58743014-aaa8-4d53-bcab-68041a59a70d None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:03.500527 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:03.516644 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c6af82-ffd0-47a6-8648-69705a26cb36 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:03.551966 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ecfd34-c64d-4bb4-833e-006a16d4a71b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:03.552551 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ecfd34-c64d-4bb4-833e-006a16d4a71b None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:03.552852 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ecfd34-c64d-4bb4-833e-006a16d4a71b None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:03.553120 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ecfd34-c64d-4bb4-833e-006a16d4a71b None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:03.553852 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ecfd34-c64d-4bb4-833e-006a16d4a71b None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:21:03.554164 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ecfd34-c64d-4bb4-833e-006a16d4a71b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:03.555582 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ecfd34-c64d-4bb4-833e-006a16d4a71b None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:03.575725 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 537/1068] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:21:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:03.578956 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ecfd34-c64d-4bb4-833e-006a16d4a71b None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:03.590694 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58743014-aaa8-4d53-bcab-68041a59a70d None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:03.617050 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:03.631171 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87868825-9249-47f9-854d-4138f78c8913 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:03.631853 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87868825-9249-47f9-854d-4138f78c8913 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:03.632145 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87868825-9249-47f9-854d-4138f78c8913 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:03.632441 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87868825-9249-47f9-854d-4138f78c8913 None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:03.633184 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87868825-9249-47f9-854d-4138f78c8913 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:21:03.633500 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87868825-9249-47f9-854d-4138f78c8913 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:03.635578 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87868825-9249-47f9-854d-4138f78c8913 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:03.652053 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 532/1069] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:21:03 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:21:03.653343 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87868825-9249-47f9-854d-4138f78c8913 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:03.694005 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ecfd34-c64d-4bb4-833e-006a16d4a71b None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:03.735751 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910e3048-cb84-4208-b0d5-6cf576e11a9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:03.736556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910e3048-cb84-4208-b0d5-6cf576e11a9f None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:03.736844 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910e3048-cb84-4208-b0d5-6cf576e11a9f None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:03.737181 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910e3048-cb84-4208-b0d5-6cf576e11a9f None admin] PATH_INFO: `/v3/users/nova` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:03.745465 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-910e3048-cb84-4208-b0d5-6cf576e11a9f None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:03.746950 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-910e3048-cb84-4208-b0d5-6cf576e11a9f None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:03.758296 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87868825-9249-47f9-854d-4138f78c8913 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:03.765685 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-910e3048-cb84-4208-b0d5-6cf576e11a9f None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 12 17:21:03.765685 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:21:03.765685 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:21:03.765685 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:21:03.765685 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:21:03.765685 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:21:03.765685 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:21:03.765685 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:21:03.765685 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:21:03.765685 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:21:03.765685 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:21:03.765685 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:21:03.765685 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:21:03.765685 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:21:03.765685 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:21:03.765685 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:21:03.766783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:21:03.766783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:21:03.766783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:21:03.766783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:21:03.766783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:21:03.766783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:21:03.766783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:21:03.766783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:21:03.766783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:21:03.766783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:21:03.766783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:21:03.766783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:21:03.766783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:21:03.766783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:21:03.766783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:21:03.766783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:21:03.766783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:21:03.766783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:21:03.767724 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:21:03.767724 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:21:03.767724 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:21:03.767724 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:21:03.767724 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:21:03.767724 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:21:03.767724 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:21:03.767724 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:21:03.767724 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 12 17:21:03.767724 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:21:03.767724 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 538/1070] 10.222.0.16 () {56 vars in 1093 bytes} [Tue Jul 12 17:21:03 2022] GET /identity/v3/users/nova => generated 82 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:21:03.773309 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910e3048-cb84-4208-b0d5-6cf576e11a9f None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:03.782006 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:03.798584 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-916ef412-6323-44e0-a744-015366ccef79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:03.799738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916ef412-6323-44e0-a744-015366ccef79 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:03.800268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916ef412-6323-44e0-a744-015366ccef79 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:03.800561 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916ef412-6323-44e0-a744-015366ccef79 None admin] PATH_INFO: `/v3/users/8754002d8c5a4870b0ecad29b09b107f` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:03.802518 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-916ef412-6323-44e0-a744-015366ccef79 None admin] RBAC: Authorizing `identity:get_user(user_id=8754002d8c5a4870b0ecad29b09b107f)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:03.804540 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-916ef412-6323-44e0-a744-015366ccef79 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:03.806461 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 533/1071] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:21:03 2022] GET /identity/v3/users/8754002d8c5a4870b0ecad29b09b107f => generated 285 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:03.812627 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-916ef412-6323-44e0-a744-015366ccef79 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:03.878055 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:03.886826 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910e3048-cb84-4208-b0d5-6cf576e11a9f None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:03.923636 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f12b185-9d42-4d5a-a7fe-f9ce58344d49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:03.923986 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-916ef412-6323-44e0-a744-015366ccef79 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:03.924315 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f12b185-9d42-4d5a-a7fe-f9ce58344d49 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:03.924600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f12b185-9d42-4d5a-a7fe-f9ce58344d49 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:03.924880 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f12b185-9d42-4d5a-a7fe-f9ce58344d49 None admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:03.925723 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f12b185-9d42-4d5a-a7fe-f9ce58344d49 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:21:03.926041 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f12b185-9d42-4d5a-a7fe-f9ce58344d49 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:03.927505 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f12b185-9d42-4d5a-a7fe-f9ce58344d49 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:03.954621 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 539/1072] 10.222.0.16 () {56 vars in 1133 bytes} [Tue Jul 12 17:21:03 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:21:03.960228 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f12b185-9d42-4d5a-a7fe-f9ce58344d49 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:03.973638 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90c1dd9-5228-4eb0-a104-803d04843a28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:03.975120 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90c1dd9-5228-4eb0-a104-803d04843a28 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:03.979673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90c1dd9-5228-4eb0-a104-803d04843a28 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:03.979673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90c1dd9-5228-4eb0-a104-803d04843a28 None admin] PATH_INFO: `/v3/projects/962bcc233b6f45b4a6adc14a7efd4ccf` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:03.979673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f90c1dd9-5228-4eb0-a104-803d04843a28 None admin] RBAC: Authorizing `identity:get_project(project_id=962bcc233b6f45b4a6adc14a7efd4ccf)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:03.979673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f90c1dd9-5228-4eb0-a104-803d04843a28 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:03.981173 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 534/1073] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:21:03 2022] GET /identity/v3/projects/962bcc233b6f45b4a6adc14a7efd4ccf => generated 311 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:21:03.987861 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90c1dd9-5228-4eb0-a104-803d04843a28 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:04.035913 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:04.075528 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f12b185-9d42-4d5a-a7fe-f9ce58344d49 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:04.095766 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90c1dd9-5228-4eb0-a104-803d04843a28 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:04.113226 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7773b43-b166-4a3f-9ff2-a39ea80fc3c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:04.114061 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7773b43-b166-4a3f-9ff2-a39ea80fc3c6 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:04.114422 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7773b43-b166-4a3f-9ff2-a39ea80fc3c6 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:04.114743 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7773b43-b166-4a3f-9ff2-a39ea80fc3c6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:04.120913 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:04.125933 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7773b43-b166-4a3f-9ff2-a39ea80fc3c6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:04.127431 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7773b43-b166-4a3f-9ff2-a39ea80fc3c6 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:04.133062 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9a4e51-52f5-4da9-a50d-c74349741cb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:04.138495 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9a4e51-52f5-4da9-a50d-c74349741cb9 None admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:04.138495 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9a4e51-52f5-4da9-a50d-c74349741cb9 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:04.138495 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9a4e51-52f5-4da9-a50d-c74349741cb9 None admin] PATH_INFO: `/v3/projects/962bcc233b6f45b4a6adc14a7efd4ccf/users/8754002d8c5a4870b0ecad29b09b107f/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:04.138495 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a9a4e51-52f5-4da9-a50d-c74349741cb9 None admin] RBAC: Authorizing `identity:create_grant(project_id=962bcc233b6f45b4a6adc14a7efd4ccf, user_id=8754002d8c5a4870b0ecad29b09b107f, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:04.138495 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a9a4e51-52f5-4da9-a50d-c74349741cb9 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:04.145399 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-f7773b43-b166-4a3f-9ff2-a39ea80fc3c6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:21:04.145399 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:21:04.145399 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:21:04.145399 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:21:04.145399 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:21:04.145399 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:21:04.145399 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:21:04.145399 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:21:04.145399 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:21:04.145399 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:21:04.145399 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:21:04.145399 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:21:04.145399 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:21:04.145399 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:21:04.145399 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:21:04.146351 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:21:04.146351 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:21:04.146351 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:21:04.146351 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:21:04.146351 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:21:04.146351 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:21:04.146351 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:21:04.146351 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:21:04.146351 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:21:04.146351 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:21:04.146351 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:21:04.146351 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:21:04.146351 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:21:04.146351 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:21:04.146351 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:21:04.146351 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:21:04.146351 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:21:04.147561 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:21:04.147561 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:21:04.147561 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:21:04.147561 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:21:04.147561 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 540/1074] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:21:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:21:04.152436 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7773b43-b166-4a3f-9ff2-a39ea80fc3c6 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:04.227076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 535/1075] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Jul 12 17:21:03 2022] PUT /identity/v3/projects/962bcc233b6f45b4a6adc14a7efd4ccf/users/8754002d8c5a4870b0ecad29b09b107f/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:21:04.258578 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7773b43-b166-4a3f-9ff2-a39ea80fc3c6 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:04.278106 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:04.298855 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097aadb2-f78f-43d8-8b83-0b8491452952 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:04.299725 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097aadb2-f78f-43d8-8b83-0b8491452952 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:04.300080 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097aadb2-f78f-43d8-8b83-0b8491452952 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:04.300428 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097aadb2-f78f-43d8-8b83-0b8491452952 None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:04.301243 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-097aadb2-f78f-43d8-8b83-0b8491452952 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:21:04.301624 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-097aadb2-f78f-43d8-8b83-0b8491452952 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:04.303129 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-097aadb2-f78f-43d8-8b83-0b8491452952 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:04.321774 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 541/1076] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:21:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:21:04.323475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9a4e51-52f5-4da9-a50d-c74349741cb9 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:04.435310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9a4e51-52f5-4da9-a50d-c74349741cb9 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:04.474373 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0bb75a-ffff-47a3-81b0-0c6a53e0b984 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:04.475273 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0bb75a-ffff-47a3-81b0-0c6a53e0b984 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:04.475574 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0bb75a-ffff-47a3-81b0-0c6a53e0b984 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:04.475846 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0bb75a-ffff-47a3-81b0-0c6a53e0b984 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:04.484156 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac0bb75a-ffff-47a3-81b0-0c6a53e0b984 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:04.485598 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac0bb75a-ffff-47a3-81b0-0c6a53e0b984 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:04.500151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-ac0bb75a-ffff-47a3-81b0-0c6a53e0b984 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:21:04.500151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:21:04.500151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:21:04.500151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:21:04.500151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:21:04.500151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:21:04.500151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:21:04.500151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:21:04.500151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:21:04.500151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:21:04.500151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:21:04.500151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:21:04.500151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:21:04.500151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:21:04.500151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:21:04.501339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:21:04.501339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:21:04.501339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:21:04.501339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:21:04.501339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:21:04.501339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:21:04.501339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:21:04.501339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:21:04.501339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:21:04.501339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:21:04.501339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:21:04.501339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:21:04.501339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:21:04.501339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:21:04.501339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:21:04.501339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:21:04.501339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:21:04.502583 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:21:04.502583 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:21:04.502583 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:21:04.502583 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:21:04.502583 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:21:04.502583 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:21:04.502583 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:21:04.502583 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:21:04.502583 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 536/1077] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:21:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:21:04.509345 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097aadb2-f78f-43d8-8b83-0b8491452952 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:04.624121 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097aadb2-f78f-43d8-8b83-0b8491452952 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:04.663978 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-043f2e79-1268-4d01-82a6-c7925cd8697d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:04.664631 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043f2e79-1268-4d01-82a6-c7925cd8697d None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:04.664915 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043f2e79-1268-4d01-82a6-c7925cd8697d None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:04.665193 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043f2e79-1268-4d01-82a6-c7925cd8697d None admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:04.666632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-043f2e79-1268-4d01-82a6-c7925cd8697d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:21:04.666952 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-043f2e79-1268-4d01-82a6-c7925cd8697d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:04.668395 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-043f2e79-1268-4d01-82a6-c7925cd8697d None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:04.686000 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 542/1078] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Jul 12 17:21:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:21:04.687600 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0bb75a-ffff-47a3-81b0-0c6a53e0b984 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:04.737098 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:04.795948 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0bb75a-ffff-47a3-81b0-0c6a53e0b984 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:04.830742 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f217757e-4360-4c83-bfa7-c39912f96dc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:04.831625 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f217757e-4360-4c83-bfa7-c39912f96dc9 None admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:04.831916 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f217757e-4360-4c83-bfa7-c39912f96dc9 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:04.832196 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f217757e-4360-4c83-bfa7-c39912f96dc9 None admin] PATH_INFO: `/v3/projects/fd4ed2fe037c4fb7ba49c724eb9414c2/users/9012d5b88f554dc596582a7693c2a72a/roles/feb6a96f7b6c4f0a8d64be226f85ee6c` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:04.834376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f217757e-4360-4c83-bfa7-c39912f96dc9 None admin] RBAC: Authorizing `identity:create_grant(project_id=fd4ed2fe037c4fb7ba49c724eb9414c2, user_id=9012d5b88f554dc596582a7693c2a72a, role_id=feb6a96f7b6c4f0a8d64be226f85ee6c)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:04.835833 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f217757e-4360-4c83-bfa7-c39912f96dc9 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:04.902591 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 537/1079] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Jul 12 17:21:04 2022] PUT /identity/v3/projects/fd4ed2fe037c4fb7ba49c724eb9414c2/users/9012d5b88f554dc596582a7693c2a72a/roles/feb6a96f7b6c4f0a8d64be226f85ee6c => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:21:04.941245 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1922513-e735-45cb-9d4e-2df3c8cf5fad None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:04.941245 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1922513-e735-45cb-9d4e-2df3c8cf5fad None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:04.941245 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1922513-e735-45cb-9d4e-2df3c8cf5fad None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:04.941245 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 543/1080] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:21:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:21:04.944134 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61497745-2770-4e2e-ab78-702fb9bbeb2a None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:04.944438 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61497745-2770-4e2e-ab78-702fb9bbeb2a None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:04.944757 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61497745-2770-4e2e-ab78-702fb9bbeb2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:04.964886 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-61497745-2770-4e2e-ab78-702fb9bbeb2a None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:21:04.966164 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-61497745-2770-4e2e-ab78-702fb9bbeb2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:21:05.043482 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 538/1081] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:21:04 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:21:05.066630 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1922513-e735-45cb-9d4e-2df3c8cf5fad None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:05.106048 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:05.175905 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1922513-e735-45cb-9d4e-2df3c8cf5fad None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:05.212463 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4dc5ce-8350-4c41-9ec2-650c48d015b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:05.213717 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4dc5ce-8350-4c41-9ec2-650c48d015b5 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:05.214039 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4dc5ce-8350-4c41-9ec2-650c48d015b5 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:05.214285 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4dc5ce-8350-4c41-9ec2-650c48d015b5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:05.284852 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f4dc5ce-8350-4c41-9ec2-650c48d015b5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:05.286581 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f4dc5ce-8350-4c41-9ec2-650c48d015b5 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:05.321023 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:05.392627 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 544/1082] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:21:05 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 12 17:21:05.398451 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61497745-2770-4e2e-ab78-702fb9bbeb2a None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:05.431438 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:05.502798 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61497745-2770-4e2e-ab78-702fb9bbeb2a None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:05.537468 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20143df-6b6d-4518-9771-9933b39c959e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:05.538592 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20143df-6b6d-4518-9771-9933b39c959e None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:05.538882 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20143df-6b6d-4518-9771-9933b39c959e None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:05.539166 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20143df-6b6d-4518-9771-9933b39c959e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:05.550063 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d20143df-6b6d-4518-9771-9933b39c959e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:05.551558 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d20143df-6b6d-4518-9771-9933b39c959e None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:05.569878 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-d20143df-6b6d-4518-9771-9933b39c959e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:21:05.569878 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:21:05.569878 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:21:05.569878 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:21:05.569878 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:21:05.569878 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:21:05.569878 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:21:05.569878 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:21:05.569878 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:21:05.569878 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:21:05.569878 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:21:05.569878 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:21:05.569878 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:21:05.569878 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:21:05.569878 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:21:05.571149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:21:05.571149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:21:05.571149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:21:05.571149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:21:05.571149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:21:05.571149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:21:05.571149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:21:05.571149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:21:05.571149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:21:05.571149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:21:05.571149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:21:05.571149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:21:05.571149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:21:05.571149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:21:05.571149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:21:05.571149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:21:05.571149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:21:05.572958 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:21:05.572958 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:21:05.572958 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:21:05.572958 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:21:05.572958 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 539/1083] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:21:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:21:05.579215 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4dc5ce-8350-4c41-9ec2-650c48d015b5 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:05.684612 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4dc5ce-8350-4c41-9ec2-650c48d015b5 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:05.719506 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8a9097-075e-419f-ba6c-53c862fe1ca9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:05.720167 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8a9097-075e-419f-ba6c-53c862fe1ca9 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:05.720450 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8a9097-075e-419f-ba6c-53c862fe1ca9 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:05.720725 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8a9097-075e-419f-ba6c-53c862fe1ca9 None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:05.721454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8a9097-075e-419f-ba6c-53c862fe1ca9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:21:05.721764 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8a9097-075e-419f-ba6c-53c862fe1ca9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:05.723135 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8a9097-075e-419f-ba6c-53c862fe1ca9 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:05.725739 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:05.736472 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 545/1084] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:21:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:06.307306 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195de545-8fcd-4be8-9fa7-aae2ef7cc695 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:06.307597 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195de545-8fcd-4be8-9fa7-aae2ef7cc695 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:06.307878 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195de545-8fcd-4be8-9fa7-aae2ef7cc695 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:06.309414 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 540/1085] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:21:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:21:06.314416 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749b1a6d-7393-45c7-88db-b9a9a9ad4359 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:06.314913 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749b1a6d-7393-45c7-88db-b9a9a9ad4359 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:06.315024 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749b1a6d-7393-45c7-88db-b9a9a9ad4359 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:06.332312 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-749b1a6d-7393-45c7-88db-b9a9a9ad4359 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:21:06.333498 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-749b1a6d-7393-45c7-88db-b9a9a9ad4359 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:21:06.400488 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 546/1086] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:21:06 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:21:06.424185 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195de545-8fcd-4be8-9fa7-aae2ef7cc695 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:06.521389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:06.538218 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195de545-8fcd-4be8-9fa7-aae2ef7cc695 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:06.576194 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0264ff2-e476-428e-b9f7-efbdd2ecc19d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:06.576930 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0264ff2-e476-428e-b9f7-efbdd2ecc19d None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:06.577221 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0264ff2-e476-428e-b9f7-efbdd2ecc19d None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:06.577506 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0264ff2-e476-428e-b9f7-efbdd2ecc19d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:06.588854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0264ff2-e476-428e-b9f7-efbdd2ecc19d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:21:06.596701 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-b0264ff2-e476-428e-b9f7-efbdd2ecc19d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:21:06.596701 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:21:06.596701 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:21:06.596701 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:21:06.596701 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:21:06.596701 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:21:06.596701 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:21:06.596701 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:21:06.596701 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:21:06.596701 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:21:06.596701 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:21:06.596701 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:21:06.596701 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:21:06.596701 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:21:06.596701 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:21:06.596701 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:21:06.597817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:21:06.597817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:21:06.597817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:21:06.597817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:21:06.597817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:21:06.597817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:21:06.597817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:21:06.597817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:21:06.597817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:21:06.597817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:21:06.597817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:21:06.597817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:21:06.597817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:21:06.597817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:21:06.597817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:21:06.597817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:21:06.599042 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:21:06.599042 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:21:06.599042 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:21:06.599042 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:21:06.599042 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:21:06.599042 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:21:06.599042 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:21:06.599042 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:21:06.599042 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:21:06.599042 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:21:06.599042 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:21:06.599042 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:21:06.602738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 541/1087] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Jul 12 17:21:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:21:06.604922 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749b1a6d-7393-45c7-88db-b9a9a9ad4359 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:06.707432 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749b1a6d-7393-45c7-88db-b9a9a9ad4359 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:06.735467 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:06.744304 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec2a297-97fd-4552-b8f6-8a84904afe3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:06.746979 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec2a297-97fd-4552-b8f6-8a84904afe3f None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:06.746979 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec2a297-97fd-4552-b8f6-8a84904afe3f None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:06.746979 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec2a297-97fd-4552-b8f6-8a84904afe3f None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:06.746979 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec2a297-97fd-4552-b8f6-8a84904afe3f None admin] RBAC: Adding query filter params (name=admin) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:21:06.746979 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec2a297-97fd-4552-b8f6-8a84904afe3f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:06.748010 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec2a297-97fd-4552-b8f6-8a84904afe3f None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:06.756299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 547/1088] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:21:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:06.762457 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0264ff2-e476-428e-b9f7-efbdd2ecc19d None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:06.865088 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0264ff2-e476-428e-b9f7-efbdd2ecc19d None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:06.898199 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1812ba39-aa0e-4598-b1ee-f1cf1a1b397e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:06.899113 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1812ba39-aa0e-4598-b1ee-f1cf1a1b397e None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:06.900996 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1812ba39-aa0e-4598-b1ee-f1cf1a1b397e None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:06.900996 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1812ba39-aa0e-4598-b1ee-f1cf1a1b397e None admin] PATH_INFO: `/v3/users/8754002d8c5a4870b0ecad29b09b107f` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:06.901586 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1812ba39-aa0e-4598-b1ee-f1cf1a1b397e None admin] RBAC: Authorizing `identity:get_user(user_id=8754002d8c5a4870b0ecad29b09b107f)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:06.903065 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1812ba39-aa0e-4598-b1ee-f1cf1a1b397e None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:06.904723 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 542/1089] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:21:06 2022] GET /identity/v3/users/8754002d8c5a4870b0ecad29b09b107f => generated 285 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:06.910709 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec2a297-97fd-4552-b8f6-8a84904afe3f None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:06.959789 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637ba62f-8f26-4b24-b7f9-00b8a1e6514d None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:06.960081 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637ba62f-8f26-4b24-b7f9-00b8a1e6514d None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:06.960370 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637ba62f-8f26-4b24-b7f9-00b8a1e6514d None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:06.961889 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 543/1090] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:21:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:21:06.967208 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713bddb6-6857-4f97-bcd7-5e1bbdb6c3d8 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:06.967504 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713bddb6-6857-4f97-bcd7-5e1bbdb6c3d8 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:06.967793 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713bddb6-6857-4f97-bcd7-5e1bbdb6c3d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:06.986396 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-713bddb6-6857-4f97-bcd7-5e1bbdb6c3d8 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:21:06.987663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-713bddb6-6857-4f97-bcd7-5e1bbdb6c3d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:21:06.996012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:07.012909 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec2a297-97fd-4552-b8f6-8a84904afe3f None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:07.047485 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e6e298-962e-4618-8ec4-2614ebeed704 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:07.048133 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e6e298-962e-4618-8ec4-2614ebeed704 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:07.048437 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e6e298-962e-4618-8ec4-2614ebeed704 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:07.048712 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e6e298-962e-4618-8ec4-2614ebeed704 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:07.057771 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 544/1091] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:21:06 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 12 17:21:07.081632 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713bddb6-6857-4f97-bcd7-5e1bbdb6c3d8 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:07.104022 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:07.116510 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e6e298-962e-4618-8ec4-2614ebeed704 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:07.117907 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65e6e298-962e-4618-8ec4-2614ebeed704 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:07.183877 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713bddb6-6857-4f97-bcd7-5e1bbdb6c3d8 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:07.219496 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d74a56b-0f4d-42d6-831a-9a5bad23a3d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:07.219717 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 548/1092] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:21:06 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 12 17:21:07.220341 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d74a56b-0f4d-42d6-831a-9a5bad23a3d4 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:07.220629 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d74a56b-0f4d-42d6-831a-9a5bad23a3d4 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:07.220909 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d74a56b-0f4d-42d6-831a-9a5bad23a3d4 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:07.225997 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e6e298-962e-4618-8ec4-2614ebeed704 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:07.231957 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d74a56b-0f4d-42d6-831a-9a5bad23a3d4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:21:07.240140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-7d74a56b-0f4d-42d6-831a-9a5bad23a3d4 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 12 17:21:07.240140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:21:07.240140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:21:07.240140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:21:07.240140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:21:07.240140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:21:07.240140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:21:07.240140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:21:07.240140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:21:07.240140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:21:07.240140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:21:07.240140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:21:07.240140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:21:07.240140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:21:07.240140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:21:07.240140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:21:07.241155 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:21:07.241155 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:21:07.241155 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:21:07.241155 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:21:07.241155 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:21:07.241155 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:21:07.241155 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:21:07.241155 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:21:07.241155 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:21:07.241155 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:21:07.241155 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:21:07.241155 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:21:07.241155 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:21:07.241155 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:21:07.241155 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:21:07.241155 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:21:07.242132 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:21:07.242132 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:21:07.242132 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:21:07.242132 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:21:07.242132 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:21:07.242132 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:21:07.242132 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:21:07.242132 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:21:07.242132 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:21:07.242132 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:21:07.242132 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 12 17:21:07.242132 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:21:07.242132 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 545/1093] 10.222.0.16 () {56 vars in 1120 bytes} [Tue Jul 12 17:21:07 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:21:07.247508 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d74a56b-0f4d-42d6-831a-9a5bad23a3d4 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:07.250250 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:07.312054 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:07.324017 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e6e298-962e-4618-8ec4-2614ebeed704 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:07.350165 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d74a56b-0f4d-42d6-831a-9a5bad23a3d4 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:07.358793 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7cea5cc-cde8-4a2a-97c3-f96c89f26dda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:07.359713 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cea5cc-cde8-4a2a-97c3-f96c89f26dda None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:07.360008 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cea5cc-cde8-4a2a-97c3-f96c89f26dda None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:07.360292 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cea5cc-cde8-4a2a-97c3-f96c89f26dda None admin] PATH_INFO: `/v3/projects/962bcc233b6f45b4a6adc14a7efd4ccf` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:07.361478 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7cea5cc-cde8-4a2a-97c3-f96c89f26dda None admin] RBAC: Authorizing `identity:get_project(project_id=962bcc233b6f45b4a6adc14a7efd4ccf)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:07.362960 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7cea5cc-cde8-4a2a-97c3-f96c89f26dda None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:07.364834 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 549/1094] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:21:07 2022] GET /identity/v3/projects/962bcc233b6f45b4a6adc14a7efd4ccf => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:21:07.371166 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7cea5cc-cde8-4a2a-97c3-f96c89f26dda None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:07.386396 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33ff23e-c4ca-4481-8010-6632a86e7098 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:07.387093 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33ff23e-c4ca-4481-8010-6632a86e7098 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:07.387379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33ff23e-c4ca-4481-8010-6632a86e7098 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:07.387666 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33ff23e-c4ca-4481-8010-6632a86e7098 None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:07.388415 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d33ff23e-c4ca-4481-8010-6632a86e7098 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:21:07.388730 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d33ff23e-c4ca-4481-8010-6632a86e7098 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:07.390094 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d33ff23e-c4ca-4481-8010-6632a86e7098 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:07.398648 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 546/1095] 10.222.0.16 () {56 vars in 1115 bytes} [Tue Jul 12 17:21:07 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:21:07.405035 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33ff23e-c4ca-4481-8010-6632a86e7098 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:07.469319 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:07.473920 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7cea5cc-cde8-4a2a-97c3-f96c89f26dda None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:07.507734 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8404f41-4412-4116-a50e-b4662b0937e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:07.508245 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33ff23e-c4ca-4481-8010-6632a86e7098 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:07.508385 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8404f41-4412-4116-a50e-b4662b0937e0 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:07.508660 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8404f41-4412-4116-a50e-b4662b0937e0 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:07.509074 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8404f41-4412-4116-a50e-b4662b0937e0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:07.510398 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8404f41-4412-4116-a50e-b4662b0937e0 None admin] RBAC: Adding query filter params (role.id=eb05bf286c894ea9bf44a6c023a4323b, scope.project.id=962bcc233b6f45b4a6adc14a7efd4ccf, user.id=8754002d8c5a4870b0ecad29b09b107f) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:21:07.510740 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8404f41-4412-4116-a50e-b4662b0937e0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:07.512173 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8404f41-4412-4116-a50e-b4662b0937e0 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:07.518241 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 550/1096] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:21:07 2022] GET /identity/v3/role_assignments?user.id=8754002d8c5a4870b0ecad29b09b107f&scope.project.id=962bcc233b6f45b4a6adc14a7efd4ccf&role.id=eb05bf286c894ea9bf44a6c023a4323b => generated 608 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:07.546802 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18f37d7-b6e4-4da5-82d0-bb799823b063 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:07.547628 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18f37d7-b6e4-4da5-82d0-bb799823b063 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:07.547921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18f37d7-b6e4-4da5-82d0-bb799823b063 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:07.548204 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18f37d7-b6e4-4da5-82d0-bb799823b063 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:07.560867 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d18f37d7-b6e4-4da5-82d0-bb799823b063 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:07.562347 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d18f37d7-b6e4-4da5-82d0-bb799823b063 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:07.584149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-d18f37d7-b6e4-4da5-82d0-bb799823b063 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:21:07.584149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:21:07.584149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:21:07.584149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:21:07.584149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:21:07.584149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:21:07.584149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:21:07.584149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:21:07.584149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:21:07.584149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:21:07.584149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:21:07.584149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:21:07.584149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:21:07.584149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:21:07.584149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:21:07.585310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:21:07.585310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:21:07.585310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:21:07.585310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:21:07.585310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:21:07.585310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:21:07.585310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:21:07.585310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:21:07.585310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:21:07.585310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:21:07.585310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:21:07.585310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:21:07.585310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:21:07.585310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:21:07.585310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:21:07.585310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:21:07.585310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:21:07.586381 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:21:07.586381 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:21:07.586381 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:21:07.586381 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:21:07.586381 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 547/1097] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:21:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:21:07.592372 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8404f41-4412-4116-a50e-b4662b0937e0 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:07.617458 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:07.704445 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8404f41-4412-4116-a50e-b4662b0937e0 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:07.739438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1ca779-0560-4b59-bbfa-83f5e5a14c82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:07.740100 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1ca779-0560-4b59-bbfa-83f5e5a14c82 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:07.740384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1ca779-0560-4b59-bbfa-83f5e5a14c82 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:07.740660 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1ca779-0560-4b59-bbfa-83f5e5a14c82 None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:07.741380 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1ca779-0560-4b59-bbfa-83f5e5a14c82 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:21:07.741691 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1ca779-0560-4b59-bbfa-83f5e5a14c82 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:07.743080 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1ca779-0560-4b59-bbfa-83f5e5a14c82 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:07.754367 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 551/1098] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:21:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:07.760562 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18f37d7-b6e4-4da5-82d0-bb799823b063 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:07.862952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18f37d7-b6e4-4da5-82d0-bb799823b063 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:07.879622 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:07.897281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9acb0a8-acb0-4a3b-9a1a-ef04152be76d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:07.898072 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9acb0a8-acb0-4a3b-9a1a-ef04152be76d None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:07.898362 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9acb0a8-acb0-4a3b-9a1a-ef04152be76d None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:07.898688 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9acb0a8-acb0-4a3b-9a1a-ef04152be76d None admin] PATH_INFO: `/v3/users/nova` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:07.906615 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9acb0a8-acb0-4a3b-9a1a-ef04152be76d None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:07.908060 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9acb0a8-acb0-4a3b-9a1a-ef04152be76d None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:07.923381 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-b9acb0a8-acb0-4a3b-9a1a-ef04152be76d None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 12 17:21:07.923381 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:21:07.923381 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:21:07.923381 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:21:07.923381 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:21:07.923381 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:21:07.923381 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:21:07.923381 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:21:07.923381 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:21:07.923381 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:21:07.923381 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:21:07.923381 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:21:07.923381 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:21:07.923381 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:21:07.923381 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:21:07.923381 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:21:07.924407 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:21:07.924407 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:21:07.924407 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:21:07.924407 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:21:07.924407 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:21:07.924407 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:21:07.924407 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:21:07.924407 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:21:07.924407 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:21:07.924407 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:21:07.924407 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:21:07.924407 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:21:07.924407 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:21:07.924407 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:21:07.924407 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:21:07.924407 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:21:07.924407 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:21:07.924407 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:21:07.925695 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:21:07.925695 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:21:07.925695 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:21:07.925695 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:21:07.925695 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:21:07.925695 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:21:07.925695 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:21:07.925695 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:21:07.925695 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 12 17:21:07.925695 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:21:07.925695 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 548/1099] 10.222.0.16 () {56 vars in 1093 bytes} [Tue Jul 12 17:21:07 2022] GET /identity/v3/users/nova => generated 82 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:21:07.930283 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1ca779-0560-4b59-bbfa-83f5e5a14c82 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:08.013858 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:08.032832 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1ca779-0560-4b59-bbfa-83f5e5a14c82 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:08.067967 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1ed919-7f5c-4b5f-acc0-22b683421354 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:08.068642 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1ed919-7f5c-4b5f-acc0-22b683421354 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:08.068936 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1ed919-7f5c-4b5f-acc0-22b683421354 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:08.069219 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1ed919-7f5c-4b5f-acc0-22b683421354 None admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:08.070065 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1ed919-7f5c-4b5f-acc0-22b683421354 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:21:08.070464 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1ed919-7f5c-4b5f-acc0-22b683421354 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:08.071975 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1ed919-7f5c-4b5f-acc0-22b683421354 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:08.089616 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 552/1100] 10.222.0.16 () {56 vars in 1133 bytes} [Tue Jul 12 17:21:07 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:21:08.095266 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9acb0a8-acb0-4a3b-9a1a-ef04152be76d None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:08.196703 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9acb0a8-acb0-4a3b-9a1a-ef04152be76d None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:08.231048 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b6b852-f0dd-4258-b7b9-a59e10748e3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:08.231696 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b6b852-f0dd-4258-b7b9-a59e10748e3e None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:08.231982 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b6b852-f0dd-4258-b7b9-a59e10748e3e None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:08.232259 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b6b852-f0dd-4258-b7b9-a59e10748e3e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:08.276809 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:08.300739 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b6b852-f0dd-4258-b7b9-a59e10748e3e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:08.302252 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b6b852-f0dd-4258-b7b9-a59e10748e3e None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:08.406950 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 549/1101] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:21:08 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 12 17:21:08.408226 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1ed919-7f5c-4b5f-acc0-22b683421354 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:08.507118 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1ed919-7f5c-4b5f-acc0-22b683421354 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:08.541555 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f2e9c0-8fa3-41ef-a0cb-16de259988f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:08.542281 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f2e9c0-8fa3-41ef-a0cb-16de259988f0 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:08.542608 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f2e9c0-8fa3-41ef-a0cb-16de259988f0 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:08.542892 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f2e9c0-8fa3-41ef-a0cb-16de259988f0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:08.549490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:08.553345 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f2e9c0-8fa3-41ef-a0cb-16de259988f0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:08.554824 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f2e9c0-8fa3-41ef-a0cb-16de259988f0 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:08.575438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-c0f2e9c0-8fa3-41ef-a0cb-16de259988f0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:21:08.575438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:21:08.575438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:21:08.575438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:21:08.575438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:21:08.575438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:21:08.575438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:21:08.575438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:21:08.575438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:21:08.575438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:21:08.575438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:21:08.575438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:21:08.575438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:21:08.575438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:21:08.575438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:21:08.576436 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:21:08.576436 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:21:08.576436 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:21:08.576436 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:21:08.576436 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:21:08.576436 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:21:08.576436 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:21:08.576436 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:21:08.576436 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:21:08.576436 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:21:08.576436 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:21:08.576436 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:21:08.576436 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:21:08.576436 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:21:08.576436 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:21:08.576436 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:21:08.576436 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:21:08.577342 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:21:08.577342 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:21:08.577342 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:21:08.577342 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:21:08.577342 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 553/1102] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:21:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:21:08.580295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b6b852-f0dd-4258-b7b9-a59e10748e3e None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:08.660612 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:08.684490 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b6b852-f0dd-4258-b7b9-a59e10748e3e None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:08.720330 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d590588e-150d-4ed8-ab1d-6aeb55010214 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:08.720994 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d590588e-150d-4ed8-ab1d-6aeb55010214 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:08.721281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d590588e-150d-4ed8-ab1d-6aeb55010214 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:08.721559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d590588e-150d-4ed8-ab1d-6aeb55010214 None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:08.722280 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d590588e-150d-4ed8-ab1d-6aeb55010214 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:21:08.722622 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d590588e-150d-4ed8-ab1d-6aeb55010214 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:08.724247 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d590588e-150d-4ed8-ab1d-6aeb55010214 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:08.740934 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 550/1103] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:21:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:21:08.742216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f2e9c0-8fa3-41ef-a0cb-16de259988f0 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:08.840364 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f2e9c0-8fa3-41ef-a0cb-16de259988f0 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:08.874620 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cfe449f-38b0-488c-9d65-6e63b0ffda14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:08.875409 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cfe449f-38b0-488c-9d65-6e63b0ffda14 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:08.875696 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cfe449f-38b0-488c-9d65-6e63b0ffda14 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:08.876027 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cfe449f-38b0-488c-9d65-6e63b0ffda14 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:08.882621 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cfe449f-38b0-488c-9d65-6e63b0ffda14 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:08.884054 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cfe449f-38b0-488c-9d65-6e63b0ffda14 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:08.897662 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-1cfe449f-38b0-488c-9d65-6e63b0ffda14 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:21:08.897662 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:21:08.897662 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:21:08.897662 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:21:08.897662 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:21:08.897662 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:21:08.897662 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:21:08.897662 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:21:08.897662 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:21:08.897662 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:21:08.897662 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:21:08.897662 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:21:08.897662 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:21:08.897662 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:21:08.897662 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:21:08.898755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:21:08.898755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:21:08.898755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:21:08.898755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:21:08.898755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:21:08.898755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:21:08.898755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:21:08.898755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:21:08.898755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:21:08.898755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:21:08.898755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:21:08.898755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:21:08.898755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:21:08.898755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:21:08.898755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:21:08.898755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:21:08.898755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:21:08.899674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:21:08.899674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:21:08.899674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:21:08.899674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:21:08.899674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:21:08.899674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:21:08.899674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:21:08.899674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:21:08.899674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 554/1104] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:21:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:21:08.904905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d590588e-150d-4ed8-ab1d-6aeb55010214 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:09.050995 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d590588e-150d-4ed8-ab1d-6aeb55010214 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:09.084725 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14860bd4-6393-4f7c-8f7c-c4f39c4a6635 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:09.085383 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14860bd4-6393-4f7c-8f7c-c4f39c4a6635 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:09.085668 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14860bd4-6393-4f7c-8f7c-c4f39c4a6635 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:09.085947 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14860bd4-6393-4f7c-8f7c-c4f39c4a6635 None admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:09.086765 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14860bd4-6393-4f7c-8f7c-c4f39c4a6635 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:21:09.087077 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14860bd4-6393-4f7c-8f7c-c4f39c4a6635 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:09.088983 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14860bd4-6393-4f7c-8f7c-c4f39c4a6635 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:09.091580 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:09.106472 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 551/1105] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Jul 12 17:21:08 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:09.108697 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cfe449f-38b0-488c-9d65-6e63b0ffda14 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:09.159424 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:09.208863 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cfe449f-38b0-488c-9d65-6e63b0ffda14 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:09.243174 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c79fff-8e41-4730-b970-b9b8239c2cb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:09.243999 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c79fff-8e41-4730-b970-b9b8239c2cb6 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:09.244363 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c79fff-8e41-4730-b970-b9b8239c2cb6 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:09.244669 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c79fff-8e41-4730-b970-b9b8239c2cb6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:09.245505 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c79fff-8e41-4730-b970-b9b8239c2cb6 None admin] RBAC: Adding query filter params (role.id=feb6a96f7b6c4f0a8d64be226f85ee6c, scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2, user.id=9012d5b88f554dc596582a7693c2a72a) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:21:09.245821 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c79fff-8e41-4730-b970-b9b8239c2cb6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:09.247288 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c79fff-8e41-4730-b970-b9b8239c2cb6 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:09.253150 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 555/1106] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:21:09 2022] GET /identity/v3/role_assignments?user.id=9012d5b88f554dc596582a7693c2a72a&scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2&role.id=feb6a96f7b6c4f0a8d64be226f85ee6c => generated 608 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:21:09.608472 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab205980-3e6f-442a-9b44-41ce7482fbcf None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:09.609005 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab205980-3e6f-442a-9b44-41ce7482fbcf None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:09.609099 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab205980-3e6f-442a-9b44-41ce7482fbcf None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:09.610452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 552/1107] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:21:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:21:09.616020 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93a52d5-adbb-42d9-b232-9c5b7e1ca5f4 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:09.616477 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93a52d5-adbb-42d9-b232-9c5b7e1ca5f4 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:09.616764 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93a52d5-adbb-42d9-b232-9c5b7e1ca5f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:09.635667 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-f93a52d5-adbb-42d9-b232-9c5b7e1ca5f4 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:21:09.636638 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-f93a52d5-adbb-42d9-b232-9c5b7e1ca5f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:21:09.709560 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 556/1108] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:21:09 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:21:09.736009 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab205980-3e6f-442a-9b44-41ce7482fbcf None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:09.844158 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab205980-3e6f-442a-9b44-41ce7482fbcf None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:09.879284 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e004cdb-74bb-49e5-83cc-9467363c3b2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:09.880085 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e004cdb-74bb-49e5-83cc-9467363c3b2a None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:09.880376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e004cdb-74bb-49e5-83cc-9467363c3b2a None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:09.880661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e004cdb-74bb-49e5-83cc-9467363c3b2a None admin] PATH_INFO: `/v3/domains/default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:09.881842 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e004cdb-74bb-49e5-83cc-9467363c3b2a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:09.883313 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e004cdb-74bb-49e5-83cc-9467363c3b2a None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:09.885620 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 553/1109] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:21:09 2022] GET /identity/v3/domains/default => generated 199 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:09.891535 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93a52d5-adbb-42d9-b232-9c5b7e1ca5f4 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:09.941812 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:10.000203 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93a52d5-adbb-42d9-b232-9c5b7e1ca5f4 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:10.036302 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab5a5a2-b3f5-4836-8935-72136a5c0ea6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:10.036983 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab5a5a2-b3f5-4836-8935-72136a5c0ea6 None admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:10.037269 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab5a5a2-b3f5-4836-8935-72136a5c0ea6 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:10.037546 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab5a5a2-b3f5-4836-8935-72136a5c0ea6 None admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:10.038273 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aab5a5a2-b3f5-4836-8935-72136a5c0ea6 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:10.039789 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aab5a5a2-b3f5-4836-8935-72136a5c0ea6 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:10.141282 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 557/1110] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Jul 12 17:21:09 2022] POST /identity/v3/users => generated 286 bytes in 250 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:21:12.197448 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b733702-1e47-4606-a3be-4f1f903a9a21 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:12.198011 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b733702-1e47-4606-a3be-4f1f903a9a21 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:12.198011 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b733702-1e47-4606-a3be-4f1f903a9a21 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:12.199416 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 554/1111] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:21:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:21:12.205252 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cdc3925-070b-4d46-810c-ba973bf50f49 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:12.205675 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cdc3925-070b-4d46-810c-ba973bf50f49 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:12.205812 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cdc3925-070b-4d46-810c-ba973bf50f49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:12.223060 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-9cdc3925-070b-4d46-810c-ba973bf50f49 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:21:12.224585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-9cdc3925-070b-4d46-810c-ba973bf50f49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:21:12.290168 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 558/1112] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:21:12 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:21:12.314360 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b733702-1e47-4606-a3be-4f1f903a9a21 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:12.377962 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:12.416112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b733702-1e47-4606-a3be-4f1f903a9a21 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:12.449746 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a592994-24c6-4d0d-bbf0-99c3ef988d39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:12.450556 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a592994-24c6-4d0d-bbf0-99c3ef988d39 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:12.450846 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a592994-24c6-4d0d-bbf0-99c3ef988d39 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:12.451128 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a592994-24c6-4d0d-bbf0-99c3ef988d39 None admin] PATH_INFO: `/v3/roles/0c239663092d459e8c9724963e78da76` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:12.452627 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a592994-24c6-4d0d-bbf0-99c3ef988d39 None admin] RBAC: Authorizing `identity:get_role(role_id=0c239663092d459e8c9724963e78da76)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:12.454062 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a592994-24c6-4d0d-bbf0-99c3ef988d39 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:12.455441 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 555/1113] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:21:12 2022] GET /identity/v3/roles/0c239663092d459e8c9724963e78da76 => generated 223 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:12.461372 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cdc3925-070b-4d46-810c-ba973bf50f49 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:12.490946 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:12.560376 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cdc3925-070b-4d46-810c-ba973bf50f49 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:12.594229 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1f45b6-4085-43e7-96e5-87c90920c6f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:12.595132 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1f45b6-4085-43e7-96e5-87c90920c6f4 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:12.595432 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1f45b6-4085-43e7-96e5-87c90920c6f4 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:12.595702 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1f45b6-4085-43e7-96e5-87c90920c6f4 None admin] PATH_INFO: `/v3/users/1caed6b52b734308b9d665fd944d1790` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:12.609661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1f45b6-4085-43e7-96e5-87c90920c6f4 None admin] RBAC: Authorizing `identity:get_user(user_id=1caed6b52b734308b9d665fd944d1790)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:12.611145 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1f45b6-4085-43e7-96e5-87c90920c6f4 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:12.612847 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 559/1114] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:21:12 2022] GET /identity/v3/users/1caed6b52b734308b9d665fd944d1790 => generated 286 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:12.620077 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a592994-24c6-4d0d-bbf0-99c3ef988d39 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:12.721890 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a592994-24c6-4d0d-bbf0-99c3ef988d39 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:12.751303 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:12.756072 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b6580c-1acd-4584-8364-501dc63cb3ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:12.756754 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b6580c-1acd-4584-8364-501dc63cb3ec None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:12.757074 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b6580c-1acd-4584-8364-501dc63cb3ec None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:12.757360 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b6580c-1acd-4584-8364-501dc63cb3ec None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:12.823932 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b6580c-1acd-4584-8364-501dc63cb3ec None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:12.825477 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b6580c-1acd-4584-8364-501dc63cb3ec None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:12.923901 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 556/1115] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:21:12 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 12 17:21:12.929905 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1f45b6-4085-43e7-96e5-87c90920c6f4 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:13.019893 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:13.031445 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1f45b6-4085-43e7-96e5-87c90920c6f4 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:13.065599 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6263daa1-2f74-4b2f-bae8-243b1e8e8789 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:13.066495 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6263daa1-2f74-4b2f-bae8-243b1e8e8789 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:13.066782 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6263daa1-2f74-4b2f-bae8-243b1e8e8789 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:13.067061 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6263daa1-2f74-4b2f-bae8-243b1e8e8789 None admin] PATH_INFO: `/v3/projects/962bcc233b6f45b4a6adc14a7efd4ccf` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:13.068251 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6263daa1-2f74-4b2f-bae8-243b1e8e8789 None admin] RBAC: Authorizing `identity:get_project(project_id=962bcc233b6f45b4a6adc14a7efd4ccf)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:13.069688 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6263daa1-2f74-4b2f-bae8-243b1e8e8789 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:13.072023 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 560/1116] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:21:12 2022] GET /identity/v3/projects/962bcc233b6f45b4a6adc14a7efd4ccf => generated 311 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:13.078160 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b6580c-1acd-4584-8364-501dc63cb3ec None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:13.109256 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:13.176993 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b6580c-1acd-4584-8364-501dc63cb3ec None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:13.211034 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5898d163-a92e-497c-b4b8-1da984901223 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:13.211691 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5898d163-a92e-497c-b4b8-1da984901223 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:13.212077 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5898d163-a92e-497c-b4b8-1da984901223 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:13.212368 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5898d163-a92e-497c-b4b8-1da984901223 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:13.213204 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5898d163-a92e-497c-b4b8-1da984901223 None admin] RBAC: Adding query filter params (role.id=0c239663092d459e8c9724963e78da76, scope.project.id=962bcc233b6f45b4a6adc14a7efd4ccf, user.id=1caed6b52b734308b9d665fd944d1790) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:21:13.213525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5898d163-a92e-497c-b4b8-1da984901223 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:13.214993 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5898d163-a92e-497c-b4b8-1da984901223 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:13.220536 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 557/1117] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:21:13 2022] GET /identity/v3/role_assignments?user.id=1caed6b52b734308b9d665fd944d1790&scope.project.id=962bcc233b6f45b4a6adc14a7efd4ccf&role.id=0c239663092d459e8c9724963e78da76 => generated 259 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:15.234701 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6c4835-64a7-453c-a339-5a1858ef137d None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:15.235238 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6c4835-64a7-453c-a339-5a1858ef137d None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:15.235332 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6c4835-64a7-453c-a339-5a1858ef137d None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:15.236689 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 561/1118] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:21:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:21:15.241900 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83be0465-ecc5-46b4-85f1-5a37065a2bf3 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:15.242320 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83be0465-ecc5-46b4-85f1-5a37065a2bf3 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:15.242498 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83be0465-ecc5-46b4-85f1-5a37065a2bf3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:15.259336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-83be0465-ecc5-46b4-85f1-5a37065a2bf3 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:21:15.260545 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-83be0465-ecc5-46b4-85f1-5a37065a2bf3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:21:15.326377 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 558/1119] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:21:15 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:21:15.350227 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6c4835-64a7-453c-a339-5a1858ef137d None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:15.450805 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6c4835-64a7-453c-a339-5a1858ef137d None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:15.484238 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e033b4-0a44-4c44-bff2-63cfcaff0464 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:15.485135 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e033b4-0a44-4c44-bff2-63cfcaff0464 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:15.485369 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e033b4-0a44-4c44-bff2-63cfcaff0464 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:15.485648 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e033b4-0a44-4c44-bff2-63cfcaff0464 None admin] PATH_INFO: `/v3/roles/0c239663092d459e8c9724963e78da76` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:15.487156 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43e033b4-0a44-4c44-bff2-63cfcaff0464 None admin] RBAC: Authorizing `identity:get_role(role_id=0c239663092d459e8c9724963e78da76)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:15.488518 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43e033b4-0a44-4c44-bff2-63cfcaff0464 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:15.489906 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 562/1120] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:21:15 2022] GET /identity/v3/roles/0c239663092d459e8c9724963e78da76 => generated 223 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:15.495797 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83be0465-ecc5-46b4-85f1-5a37065a2bf3 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:15.522750 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:15.594578 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83be0465-ecc5-46b4-85f1-5a37065a2bf3 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:15.631049 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9f8072-5837-4160-b73f-48c0fb755167 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:15.632246 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9f8072-5837-4160-b73f-48c0fb755167 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:15.632246 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9f8072-5837-4160-b73f-48c0fb755167 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:15.632597 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9f8072-5837-4160-b73f-48c0fb755167 None admin] PATH_INFO: `/v3/users/1caed6b52b734308b9d665fd944d1790` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:15.633856 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9f8072-5837-4160-b73f-48c0fb755167 None admin] RBAC: Authorizing `identity:get_user(user_id=1caed6b52b734308b9d665fd944d1790)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:15.635511 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9f8072-5837-4160-b73f-48c0fb755167 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:15.636962 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 559/1121] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:21:15 2022] GET /identity/v3/users/1caed6b52b734308b9d665fd944d1790 => generated 286 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:15.642996 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e033b4-0a44-4c44-bff2-63cfcaff0464 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:15.669159 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:15.743100 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e033b4-0a44-4c44-bff2-63cfcaff0464 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:15.776875 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e5a6254-f580-4c00-8c63-9d1374f4aa21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:15.777725 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5a6254-f580-4c00-8c63-9d1374f4aa21 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:15.778009 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5a6254-f580-4c00-8c63-9d1374f4aa21 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:15.778284 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5a6254-f580-4c00-8c63-9d1374f4aa21 None admin] PATH_INFO: `/v3/projects/962bcc233b6f45b4a6adc14a7efd4ccf` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:15.779489 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e5a6254-f580-4c00-8c63-9d1374f4aa21 None admin] RBAC: Authorizing `identity:get_project(project_id=962bcc233b6f45b4a6adc14a7efd4ccf)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:15.780899 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e5a6254-f580-4c00-8c63-9d1374f4aa21 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:15.782628 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 563/1122] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:21:15 2022] GET /identity/v3/projects/962bcc233b6f45b4a6adc14a7efd4ccf => generated 311 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:15.788783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9f8072-5837-4160-b73f-48c0fb755167 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:15.879541 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:15.886143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9f8072-5837-4160-b73f-48c0fb755167 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:15.919851 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-043df398-c2fe-4b58-95c8-43696fdd96d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:15.920729 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043df398-c2fe-4b58-95c8-43696fdd96d3 None admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:15.921013 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043df398-c2fe-4b58-95c8-43696fdd96d3 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:15.921288 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043df398-c2fe-4b58-95c8-43696fdd96d3 None admin] PATH_INFO: `/v3/projects/962bcc233b6f45b4a6adc14a7efd4ccf/users/1caed6b52b734308b9d665fd944d1790/roles/0c239663092d459e8c9724963e78da76` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:15.923468 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-043df398-c2fe-4b58-95c8-43696fdd96d3 None admin] RBAC: Authorizing `identity:create_grant(project_id=962bcc233b6f45b4a6adc14a7efd4ccf, user_id=1caed6b52b734308b9d665fd944d1790, role_id=0c239663092d459e8c9724963e78da76)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:15.924944 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-043df398-c2fe-4b58-95c8-43696fdd96d3 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:16.023340 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 560/1123] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Jul 12 17:21:15 2022] PUT /identity/v3/projects/962bcc233b6f45b4a6adc14a7efd4ccf/users/1caed6b52b734308b9d665fd944d1790/roles/0c239663092d459e8c9724963e78da76 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:21:18.052359 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3af72f-57e3-496a-94da-dae5c437ffb2 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:18.052887 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3af72f-57e3-496a-94da-dae5c437ffb2 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:18.053000 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec3af72f-57e3-496a-94da-dae5c437ffb2 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:18.054336 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 564/1124] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:21:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:21:18.060472 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426eadda-02b3-46ac-9631-e5642fdd9b31 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:18.060856 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426eadda-02b3-46ac-9631-e5642fdd9b31 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:18.061040 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426eadda-02b3-46ac-9631-e5642fdd9b31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:18.078336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-426eadda-02b3-46ac-9631-e5642fdd9b31 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:21:18.079556 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-426eadda-02b3-46ac-9631-e5642fdd9b31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:21:18.145700 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 561/1125] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:21:18 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:21:18.169348 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec3af72f-57e3-496a-94da-dae5c437ffb2 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:18.260037 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:18.269230 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec3af72f-57e3-496a-94da-dae5c437ffb2 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:18.302474 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d56b53-b140-48d1-a882-f03219ce2d50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:18.303291 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d56b53-b140-48d1-a882-f03219ce2d50 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:18.303577 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d56b53-b140-48d1-a882-f03219ce2d50 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:18.303855 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d56b53-b140-48d1-a882-f03219ce2d50 None admin] PATH_INFO: `/v3/roles/0c239663092d459e8c9724963e78da76` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:18.305876 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d56b53-b140-48d1-a882-f03219ce2d50 None admin] RBAC: Authorizing `identity:get_role(role_id=0c239663092d459e8c9724963e78da76)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:18.307437 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d56b53-b140-48d1-a882-f03219ce2d50 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:18.308779 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 565/1126] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:21:18 2022] GET /identity/v3/roles/0c239663092d459e8c9724963e78da76 => generated 223 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:18.314765 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426eadda-02b3-46ac-9631-e5642fdd9b31 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:18.400153 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:18.416231 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-426eadda-02b3-46ac-9631-e5642fdd9b31 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:18.450155 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c10131a-8d9a-4011-810b-f7a2c1bea409 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:18.451093 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c10131a-8d9a-4011-810b-f7a2c1bea409 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:18.451380 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c10131a-8d9a-4011-810b-f7a2c1bea409 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:18.451661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c10131a-8d9a-4011-810b-f7a2c1bea409 None admin] PATH_INFO: `/v3/users/1caed6b52b734308b9d665fd944d1790` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:18.452871 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c10131a-8d9a-4011-810b-f7a2c1bea409 None admin] RBAC: Authorizing `identity:get_user(user_id=1caed6b52b734308b9d665fd944d1790)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:18.454303 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c10131a-8d9a-4011-810b-f7a2c1bea409 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:18.455963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 562/1127] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:21:18 2022] GET /identity/v3/users/1caed6b52b734308b9d665fd944d1790 => generated 286 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:21:18.462112 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d56b53-b140-48d1-a882-f03219ce2d50 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:18.561416 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d56b53-b140-48d1-a882-f03219ce2d50 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:18.594831 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f9d19b-94fc-4818-abdf-add29503c075 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:18.595490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f9d19b-94fc-4818-abdf-add29503c075 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:18.595781 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f9d19b-94fc-4818-abdf-add29503c075 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:18.596064 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f9d19b-94fc-4818-abdf-add29503c075 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:18.624946 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:18.664527 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9f9d19b-94fc-4818-abdf-add29503c075 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:18.666528 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9f9d19b-94fc-4818-abdf-add29503c075 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:18.763689 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 566/1128] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:21:18 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 12 17:21:18.769693 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c10131a-8d9a-4011-810b-f7a2c1bea409 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:18.869006 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c10131a-8d9a-4011-810b-f7a2c1bea409 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:18.902399 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459881de-413a-4511-b28f-20eb074d45f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:18.903278 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459881de-413a-4511-b28f-20eb074d45f9 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:18.903571 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459881de-413a-4511-b28f-20eb074d45f9 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:18.903854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459881de-413a-4511-b28f-20eb074d45f9 None admin] PATH_INFO: `/v3/projects/962bcc233b6f45b4a6adc14a7efd4ccf` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:18.905036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459881de-413a-4511-b28f-20eb074d45f9 None admin] RBAC: Authorizing `identity:get_project(project_id=962bcc233b6f45b4a6adc14a7efd4ccf)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:18.906485 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459881de-413a-4511-b28f-20eb074d45f9 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:18.908282 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 563/1129] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:21:18 2022] GET /identity/v3/projects/962bcc233b6f45b4a6adc14a7efd4ccf => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:18.914738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f9d19b-94fc-4818-abdf-add29503c075 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:18.973769 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:19.013084 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f9d19b-94fc-4818-abdf-add29503c075 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:19.046256 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629de4db-a3a1-4254-9b19-7531aa22583a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:19.046929 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629de4db-a3a1-4254-9b19-7531aa22583a None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:19.047219 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629de4db-a3a1-4254-9b19-7531aa22583a None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:19.047498 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629de4db-a3a1-4254-9b19-7531aa22583a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:19.048323 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629de4db-a3a1-4254-9b19-7531aa22583a None admin] RBAC: Adding query filter params (role.id=0c239663092d459e8c9724963e78da76, scope.project.id=962bcc233b6f45b4a6adc14a7efd4ccf, user.id=1caed6b52b734308b9d665fd944d1790) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:21:19.048636 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629de4db-a3a1-4254-9b19-7531aa22583a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:19.050061 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629de4db-a3a1-4254-9b19-7531aa22583a None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:19.056633 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 567/1130] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:21:18 2022] GET /identity/v3/role_assignments?user.id=1caed6b52b734308b9d665fd944d1790&scope.project.id=962bcc233b6f45b4a6adc14a7efd4ccf&role.id=0c239663092d459e8c9724963e78da76 => generated 608 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:21.267806 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b518051-d050-42f7-bb5e-5c31786767c8 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:21.268372 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b518051-d050-42f7-bb5e-5c31786767c8 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:21.268372 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b518051-d050-42f7-bb5e-5c31786767c8 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:21.269783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 564/1131] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:21:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:21:21.275235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f476b30-1684-4570-853a-0bdb60f37bf5 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:21.275662 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f476b30-1684-4570-853a-0bdb60f37bf5 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:21.275796 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f476b30-1684-4570-853a-0bdb60f37bf5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:21.292261 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-8f476b30-1684-4570-853a-0bdb60f37bf5 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:21:21.293479 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-8f476b30-1684-4570-853a-0bdb60f37bf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:21:21.359491 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 568/1132] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:21:21 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:21:21.383435 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b518051-d050-42f7-bb5e-5c31786767c8 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:21.404751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:21.483315 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b518051-d050-42f7-bb5e-5c31786767c8 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:21.517239 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf23dd71-757a-4a6c-9b39-63bbd738a1ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:21.518173 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf23dd71-757a-4a6c-9b39-63bbd738a1ad None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:21.518412 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf23dd71-757a-4a6c-9b39-63bbd738a1ad None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:21.518723 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf23dd71-757a-4a6c-9b39-63bbd738a1ad None admin] PATH_INFO: `/v3/domains/default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:21.519892 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf23dd71-757a-4a6c-9b39-63bbd738a1ad None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:21.521316 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf23dd71-757a-4a6c-9b39-63bbd738a1ad None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:21.522966 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 565/1133] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:21:21 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:21.529177 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f476b30-1684-4570-853a-0bdb60f37bf5 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:21.584989 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:21.628552 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f476b30-1684-4570-853a-0bdb60f37bf5 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:21.661887 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81b81fb1-db49-4956-a2b4-4a2f3030e06b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:21.662584 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b81fb1-db49-4956-a2b4-4a2f3030e06b None admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:21.662875 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b81fb1-db49-4956-a2b4-4a2f3030e06b None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:21.663158 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b81fb1-db49-4956-a2b4-4a2f3030e06b None admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:21.663863 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81b81fb1-db49-4956-a2b4-4a2f3030e06b None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:21.665934 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81b81fb1-db49-4956-a2b4-4a2f3030e06b None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:21.786061 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 569/1134] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Jul 12 17:21:21 2022] POST /identity/v3/projects => generated 311 bytes in 257 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:21:23.846897 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f5687d-3841-4665-9814-34bafea5a033 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:23.847647 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f5687d-3841-4665-9814-34bafea5a033 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:23.847831 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f5687d-3841-4665-9814-34bafea5a033 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:23.849216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 566/1135] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:21:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:21:23.855207 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217ad04a-afdf-45c5-a927-20a54302a041 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:23.855611 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217ad04a-afdf-45c5-a927-20a54302a041 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:23.855761 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217ad04a-afdf-45c5-a927-20a54302a041 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:23.872800 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-217ad04a-afdf-45c5-a927-20a54302a041 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:21:23.873975 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-217ad04a-afdf-45c5-a927-20a54302a041 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:21:23.938678 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 570/1136] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:21:23 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:21:23.964065 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f5687d-3841-4665-9814-34bafea5a033 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:24.054140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:24.067663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f5687d-3841-4665-9814-34bafea5a033 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:24.105384 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6105e37a-33c4-4c36-aed8-a48b7b1261e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:24.106091 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6105e37a-33c4-4c36-aed8-a48b7b1261e8 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:24.106424 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6105e37a-33c4-4c36-aed8-a48b7b1261e8 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:24.106855 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6105e37a-33c4-4c36-aed8-a48b7b1261e8 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:24.108052 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6105e37a-33c4-4c36-aed8-a48b7b1261e8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:24.109478 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6105e37a-33c4-4c36-aed8-a48b7b1261e8 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:24.111153 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 567/1137] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:21:23 2022] GET /identity/v3/domains/default => generated 199 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:24.116933 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217ad04a-afdf-45c5-a927-20a54302a041 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:24.151590 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:24.214411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217ad04a-afdf-45c5-a927-20a54302a041 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:24.247540 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f45de547-ceb2-4271-8f11-8bf335859048 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:24.248206 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45de547-ceb2-4271-8f11-8bf335859048 None admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:24.248496 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45de547-ceb2-4271-8f11-8bf335859048 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:24.248773 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45de547-ceb2-4271-8f11-8bf335859048 None admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:24.249473 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f45de547-ceb2-4271-8f11-8bf335859048 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:24.250976 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f45de547-ceb2-4271-8f11-8bf335859048 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:24.385607 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 571/1138] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Jul 12 17:21:24 2022] POST /identity/v3/users => generated 286 bytes in 269 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:21:26.458345 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c8e991-cfcb-40c5-934b-3deb858cb388 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:26.458901 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c8e991-cfcb-40c5-934b-3deb858cb388 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:26.458995 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c8e991-cfcb-40c5-934b-3deb858cb388 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:26.460300 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 568/1139] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:21:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:21:26.465687 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa92ca6-ba68-47a0-a373-c11c627068af None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:26.466136 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa92ca6-ba68-47a0-a373-c11c627068af None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:26.466240 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa92ca6-ba68-47a0-a373-c11c627068af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:26.483119 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-8fa92ca6-ba68-47a0-a373-c11c627068af None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:21:26.484314 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-8fa92ca6-ba68-47a0-a373-c11c627068af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:21:26.549571 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 572/1140] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:21:26 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:21:26.573900 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c8e991-cfcb-40c5-934b-3deb858cb388 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:26.674794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c8e991-cfcb-40c5-934b-3deb858cb388 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:26.708472 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937e321f-0761-4b06-9eef-92781f3d180e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:26.709194 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937e321f-0761-4b06-9eef-92781f3d180e None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:26.709488 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937e321f-0761-4b06-9eef-92781f3d180e None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:26.709770 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937e321f-0761-4b06-9eef-92781f3d180e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:26.719885 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-937e321f-0761-4b06-9eef-92781f3d180e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:21:26.728324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-937e321f-0761-4b06-9eef-92781f3d180e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:21:26.728324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:21:26.728324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:21:26.728324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:21:26.728324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:21:26.728324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:21:26.728324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:21:26.728324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:21:26.728324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:21:26.728324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:21:26.728324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:21:26.728324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:21:26.728324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:21:26.728324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:21:26.728324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:21:26.728324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:21:26.730377 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:21:26.730377 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:21:26.730377 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:21:26.730377 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:21:26.730377 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:21:26.730377 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:21:26.730377 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:21:26.730377 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:21:26.730377 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:21:26.730377 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:21:26.730377 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:21:26.730377 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:21:26.730377 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:21:26.730377 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:21:26.730377 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:21:26.730377 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:21:26.731481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:21:26.731481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:21:26.731481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:21:26.731481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:21:26.731481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:21:26.731481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:21:26.731481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:21:26.731481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:21:26.731481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:21:26.731481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:21:26.731481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:21:26.731481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:21:26.731481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 569/1141] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Jul 12 17:21:26 2022] GET /identity/v3/roles/admin => generated 83 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:21:26.735848 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa92ca6-ba68-47a0-a373-c11c627068af None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:26.759651 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:26.834031 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa92ca6-ba68-47a0-a373-c11c627068af None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:26.868071 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c257c17-8aac-4d22-8da9-85559f25502e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:26.868733 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c257c17-8aac-4d22-8da9-85559f25502e None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:26.869019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c257c17-8aac-4d22-8da9-85559f25502e None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:26.869299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c257c17-8aac-4d22-8da9-85559f25502e None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:26.870047 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c257c17-8aac-4d22-8da9-85559f25502e None admin] RBAC: Adding query filter params (name=admin) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:21:26.870359 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c257c17-8aac-4d22-8da9-85559f25502e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:26.871751 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c257c17-8aac-4d22-8da9-85559f25502e None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:26.879772 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 573/1142] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:21:26 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:26.885802 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937e321f-0761-4b06-9eef-92781f3d180e None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:26.904091 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:26.985817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937e321f-0761-4b06-9eef-92781f3d180e None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:27.018832 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f011e1-3c31-40e6-ae5e-e1dc00784af3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:27.019722 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f011e1-3c31-40e6-ae5e-e1dc00784af3 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:27.019980 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f011e1-3c31-40e6-ae5e-e1dc00784af3 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:27.020262 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f011e1-3c31-40e6-ae5e-e1dc00784af3 None admin] PATH_INFO: `/v3/users/8c62e70fbb4d4134a70e21f79cf2ade0` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:27.033838 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7f011e1-3c31-40e6-ae5e-e1dc00784af3 None admin] RBAC: Authorizing `identity:get_user(user_id=8c62e70fbb4d4134a70e21f79cf2ade0)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:27.035327 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7f011e1-3c31-40e6-ae5e-e1dc00784af3 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:27.036984 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 570/1143] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:21:26 2022] GET /identity/v3/users/8c62e70fbb4d4134a70e21f79cf2ade0 => generated 286 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:27.043538 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c257c17-8aac-4d22-8da9-85559f25502e None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:27.126122 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:27.141127 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c257c17-8aac-4d22-8da9-85559f25502e None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:27.174820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0159cae3-4831-45a6-a9db-91fce6c5ecb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:27.175462 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0159cae3-4831-45a6-a9db-91fce6c5ecb3 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:27.175751 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0159cae3-4831-45a6-a9db-91fce6c5ecb3 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:27.176035 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0159cae3-4831-45a6-a9db-91fce6c5ecb3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:27.243073 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0159cae3-4831-45a6-a9db-91fce6c5ecb3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:27.244443 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0159cae3-4831-45a6-a9db-91fce6c5ecb3 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:27.324676 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:27.341409 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 574/1144] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:21:27 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 12 17:21:27.347666 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f011e1-3c31-40e6-ae5e-e1dc00784af3 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:27.422447 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:27.447170 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f011e1-3c31-40e6-ae5e-e1dc00784af3 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:27.480143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2764b5ee-08ec-47ef-8ad0-f0f2c818065c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:27.480988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2764b5ee-08ec-47ef-8ad0-f0f2c818065c None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:27.481267 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2764b5ee-08ec-47ef-8ad0-f0f2c818065c None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:27.481539 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2764b5ee-08ec-47ef-8ad0-f0f2c818065c None admin] PATH_INFO: `/v3/projects/61211724a3d74aaa873122b3d9d55a2e` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:27.482720 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2764b5ee-08ec-47ef-8ad0-f0f2c818065c None admin] RBAC: Authorizing `identity:get_project(project_id=61211724a3d74aaa873122b3d9d55a2e)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:27.484129 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2764b5ee-08ec-47ef-8ad0-f0f2c818065c None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:27.485794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 571/1145] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:21:27 2022] GET /identity/v3/projects/61211724a3d74aaa873122b3d9d55a2e => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:27.491881 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0159cae3-4831-45a6-a9db-91fce6c5ecb3 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:27.590296 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0159cae3-4831-45a6-a9db-91fce6c5ecb3 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:27.624954 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af5fb27-5b12-4483-a891-8e8ef5a27896 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:27.625609 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af5fb27-5b12-4483-a891-8e8ef5a27896 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:27.625897 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af5fb27-5b12-4483-a891-8e8ef5a27896 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:27.626177 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af5fb27-5b12-4483-a891-8e8ef5a27896 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:27.627037 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af5fb27-5b12-4483-a891-8e8ef5a27896 None admin] RBAC: Adding query filter params (role.id=eb05bf286c894ea9bf44a6c023a4323b, scope.project.id=61211724a3d74aaa873122b3d9d55a2e, user.id=8c62e70fbb4d4134a70e21f79cf2ade0) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:21:27.627353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af5fb27-5b12-4483-a891-8e8ef5a27896 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:27.628837 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af5fb27-5b12-4483-a891-8e8ef5a27896 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:27.634642 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 575/1146] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:21:27 2022] GET /identity/v3/role_assignments?user.id=8c62e70fbb4d4134a70e21f79cf2ade0&scope.project.id=61211724a3d74aaa873122b3d9d55a2e&role.id=eb05bf286c894ea9bf44a6c023a4323b => generated 259 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:29.662245 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae13549f-9e58-4bf1-bcb4-9a69e9d77ebb None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:29.662849 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae13549f-9e58-4bf1-bcb4-9a69e9d77ebb None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:29.662849 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae13549f-9e58-4bf1-bcb4-9a69e9d77ebb None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:29.664896 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 572/1147] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:21:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:21:29.670182 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f26189-6ad6-469c-879e-ee2a3aef95e4 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:29.670582 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f26189-6ad6-469c-879e-ee2a3aef95e4 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:29.670784 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f26189-6ad6-469c-879e-ee2a3aef95e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:29.687987 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-16f26189-6ad6-469c-879e-ee2a3aef95e4 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:21:29.689164 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-16f26189-6ad6-469c-879e-ee2a3aef95e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:21:29.733219 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:29.756028 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 576/1148] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:21:29 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:21:29.778229 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae13549f-9e58-4bf1-bcb4-9a69e9d77ebb None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:29.879426 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae13549f-9e58-4bf1-bcb4-9a69e9d77ebb None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:29.911875 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb54d45c-2a72-4eeb-b60f-7e795f9d439c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:29.912660 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb54d45c-2a72-4eeb-b60f-7e795f9d439c None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:29.912943 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb54d45c-2a72-4eeb-b60f-7e795f9d439c None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:29.913220 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb54d45c-2a72-4eeb-b60f-7e795f9d439c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:29.923622 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb54d45c-2a72-4eeb-b60f-7e795f9d439c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:21:29.931988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-eb54d45c-2a72-4eeb-b60f-7e795f9d439c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:21:29.931988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:21:29.931988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:21:29.931988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:21:29.931988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:21:29.931988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:21:29.931988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:21:29.931988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:21:29.931988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:21:29.931988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:21:29.931988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:21:29.931988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:21:29.931988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:21:29.931988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:21:29.931988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:21:29.931988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:21:29.933111 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:21:29.933111 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:21:29.933111 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:21:29.933111 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:21:29.933111 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:21:29.933111 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:21:29.933111 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:21:29.933111 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:21:29.933111 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:21:29.933111 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:21:29.933111 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:21:29.933111 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:21:29.933111 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:21:29.933111 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:21:29.933111 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:21:29.933111 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:21:29.934281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:21:29.934281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:21:29.934281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:21:29.934281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:21:29.934281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:21:29.934281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:21:29.934281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:21:29.934281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:21:29.934281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:21:29.934281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:21:29.934281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:21:29.934281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:21:29.934281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 573/1149] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Jul 12 17:21:29 2022] GET /identity/v3/roles/admin => generated 83 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:21:29.939646 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f26189-6ad6-469c-879e-ee2a3aef95e4 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:30.035973 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f26189-6ad6-469c-879e-ee2a3aef95e4 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:30.069877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b704a70c-af78-4202-acf0-ea18c68e2f77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:30.070581 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b704a70c-af78-4202-acf0-ea18c68e2f77 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:30.070871 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b704a70c-af78-4202-acf0-ea18c68e2f77 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:30.071151 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b704a70c-af78-4202-acf0-ea18c68e2f77 None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:30.071905 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b704a70c-af78-4202-acf0-ea18c68e2f77 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:21:30.072218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b704a70c-af78-4202-acf0-ea18c68e2f77 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:30.073588 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b704a70c-af78-4202-acf0-ea18c68e2f77 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:30.081642 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 577/1150] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:21:29 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:30.087942 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb54d45c-2a72-4eeb-b60f-7e795f9d439c None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:30.090787 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:30.187829 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb54d45c-2a72-4eeb-b60f-7e795f9d439c None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:30.220522 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6908a4ea-b5a9-4f54-9dce-0db57957b72b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:30.221377 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6908a4ea-b5a9-4f54-9dce-0db57957b72b None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:30.221659 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6908a4ea-b5a9-4f54-9dce-0db57957b72b None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:30.222189 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6908a4ea-b5a9-4f54-9dce-0db57957b72b None admin] PATH_INFO: `/v3/users/8c62e70fbb4d4134a70e21f79cf2ade0` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:30.223390 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6908a4ea-b5a9-4f54-9dce-0db57957b72b None admin] RBAC: Authorizing `identity:get_user(user_id=8c62e70fbb4d4134a70e21f79cf2ade0)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:30.224805 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6908a4ea-b5a9-4f54-9dce-0db57957b72b None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:30.227264 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 574/1151] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:21:30 2022] GET /identity/v3/users/8c62e70fbb4d4134a70e21f79cf2ade0 => generated 286 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:30.233247 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b704a70c-af78-4202-acf0-ea18c68e2f77 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:30.275823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:30.333471 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b704a70c-af78-4202-acf0-ea18c68e2f77 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:30.366798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aabb7fe-8d02-4e30-bd5a-d5c960e84024 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:30.367662 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aabb7fe-8d02-4e30-bd5a-d5c960e84024 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:30.367953 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aabb7fe-8d02-4e30-bd5a-d5c960e84024 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:30.368233 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aabb7fe-8d02-4e30-bd5a-d5c960e84024 None admin] PATH_INFO: `/v3/projects/61211724a3d74aaa873122b3d9d55a2e` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:30.369419 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aabb7fe-8d02-4e30-bd5a-d5c960e84024 None admin] RBAC: Authorizing `identity:get_project(project_id=61211724a3d74aaa873122b3d9d55a2e)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:30.370884 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aabb7fe-8d02-4e30-bd5a-d5c960e84024 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:30.372762 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 578/1152] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:21:30 2022] GET /identity/v3/projects/61211724a3d74aaa873122b3d9d55a2e => generated 311 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:30.379041 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6908a4ea-b5a9-4f54-9dce-0db57957b72b None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:30.448087 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:30.479137 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6908a4ea-b5a9-4f54-9dce-0db57957b72b None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:30.512198 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27d0df3-865d-4a0f-a969-4d1ee0b03087 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:30.513071 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27d0df3-865d-4a0f-a969-4d1ee0b03087 None admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:30.513357 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27d0df3-865d-4a0f-a969-4d1ee0b03087 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:30.513632 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27d0df3-865d-4a0f-a969-4d1ee0b03087 None admin] PATH_INFO: `/v3/projects/61211724a3d74aaa873122b3d9d55a2e/users/8c62e70fbb4d4134a70e21f79cf2ade0/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:30.515565 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27d0df3-865d-4a0f-a969-4d1ee0b03087 None admin] RBAC: Authorizing `identity:create_grant(project_id=61211724a3d74aaa873122b3d9d55a2e, user_id=8c62e70fbb4d4134a70e21f79cf2ade0, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:30.516984 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27d0df3-865d-4a0f-a969-4d1ee0b03087 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:30.607492 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 575/1153] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Jul 12 17:21:30 2022] PUT /identity/v3/projects/61211724a3d74aaa873122b3d9d55a2e/users/8c62e70fbb4d4134a70e21f79cf2ade0/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:21:32.640332 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75f5c65-c2eb-4007-8891-dbd4bd8aacb5 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:32.640858 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75f5c65-c2eb-4007-8891-dbd4bd8aacb5 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:32.640945 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75f5c65-c2eb-4007-8891-dbd4bd8aacb5 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:32.642255 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 579/1154] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:21:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:21:32.647545 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce689175-6ea6-4fde-a5ee-382ae4d91820 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:32.648066 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce689175-6ea6-4fde-a5ee-382ae4d91820 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:32.648156 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce689175-6ea6-4fde-a5ee-382ae4d91820 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:32.665178 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-ce689175-6ea6-4fde-a5ee-382ae4d91820 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:21:32.666371 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-ce689175-6ea6-4fde-a5ee-382ae4d91820 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:21:32.735974 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 576/1155] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:21:32 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:21:32.760947 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75f5c65-c2eb-4007-8891-dbd4bd8aacb5 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:32.864080 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75f5c65-c2eb-4007-8891-dbd4bd8aacb5 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:32.872895 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:32.896583 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df6a22f-0da5-4a6f-ad40-c95c7cbbdcfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:32.897573 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df6a22f-0da5-4a6f-ad40-c95c7cbbdcfa None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:32.898219 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df6a22f-0da5-4a6f-ad40-c95c7cbbdcfa None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:32.898774 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df6a22f-0da5-4a6f-ad40-c95c7cbbdcfa None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:32.909636 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df6a22f-0da5-4a6f-ad40-c95c7cbbdcfa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:21:32.917384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-4df6a22f-0da5-4a6f-ad40-c95c7cbbdcfa None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:21:32.917384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:21:32.917384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:21:32.917384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:21:32.917384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:21:32.917384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:21:32.917384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:21:32.917384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:21:32.917384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:21:32.917384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:21:32.917384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:21:32.917384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:21:32.917384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:21:32.917384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:21:32.917384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:21:32.917384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:21:32.918661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:21:32.918661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:21:32.918661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:21:32.918661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:21:32.918661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:21:32.918661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:21:32.918661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:21:32.918661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:21:32.918661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:21:32.918661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:21:32.918661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:21:32.918661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:21:32.918661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:21:32.918661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:21:32.918661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:21:32.918661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:21:32.919802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:21:32.919802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:21:32.919802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:21:32.919802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:21:32.919802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:21:32.919802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:21:32.919802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:21:32.919802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:21:32.919802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:21:32.919802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:21:32.919802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:21:32.919802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:21:32.919802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 580/1156] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Jul 12 17:21:32 2022] GET /identity/v3/roles/admin => generated 83 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:21:32.924717 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce689175-6ea6-4fde-a5ee-382ae4d91820 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:32.983246 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:33.022951 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce689175-6ea6-4fde-a5ee-382ae4d91820 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:33.056267 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41072347-f143-4fc8-beb4-6562f8120040 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:33.056943 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41072347-f143-4fc8-beb4-6562f8120040 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:33.057232 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41072347-f143-4fc8-beb4-6562f8120040 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:33.057513 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41072347-f143-4fc8-beb4-6562f8120040 None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:33.058259 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41072347-f143-4fc8-beb4-6562f8120040 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:21:33.058624 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41072347-f143-4fc8-beb4-6562f8120040 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:33.060100 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41072347-f143-4fc8-beb4-6562f8120040 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:33.068176 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 577/1157] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:21:32 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:33.074798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df6a22f-0da5-4a6f-ad40-c95c7cbbdcfa None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:33.173031 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df6a22f-0da5-4a6f-ad40-c95c7cbbdcfa None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:33.206741 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74621d14-1f32-4288-b3f3-110e1c646a90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:33.207603 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74621d14-1f32-4288-b3f3-110e1c646a90 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:33.207892 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74621d14-1f32-4288-b3f3-110e1c646a90 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:33.208175 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74621d14-1f32-4288-b3f3-110e1c646a90 None admin] PATH_INFO: `/v3/users/8c62e70fbb4d4134a70e21f79cf2ade0` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:33.209357 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74621d14-1f32-4288-b3f3-110e1c646a90 None admin] RBAC: Authorizing `identity:get_user(user_id=8c62e70fbb4d4134a70e21f79cf2ade0)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:33.210929 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74621d14-1f32-4288-b3f3-110e1c646a90 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:33.212585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 581/1158] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:21:33 2022] GET /identity/v3/users/8c62e70fbb4d4134a70e21f79cf2ade0 => generated 286 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:33.218410 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41072347-f143-4fc8-beb4-6562f8120040 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:33.316887 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41072347-f143-4fc8-beb4-6562f8120040 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:33.339333 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:33.351235 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-429e615c-a1a1-4057-84a5-41dae11003ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:33.352723 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429e615c-a1a1-4057-84a5-41dae11003ef None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:33.352723 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429e615c-a1a1-4057-84a5-41dae11003ef None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:33.352723 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429e615c-a1a1-4057-84a5-41dae11003ef None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:33.417986 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-429e615c-a1a1-4057-84a5-41dae11003ef None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:33.419438 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-429e615c-a1a1-4057-84a5-41dae11003ef None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:33.516178 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 578/1159] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:21:33 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 12 17:21:33.522087 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74621d14-1f32-4288-b3f3-110e1c646a90 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:33.557377 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:33.620608 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74621d14-1f32-4288-b3f3-110e1c646a90 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:33.653887 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f885986c-44c9-419f-a368-0d2d79f9f088 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:33.654773 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f885986c-44c9-419f-a368-0d2d79f9f088 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:33.655059 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f885986c-44c9-419f-a368-0d2d79f9f088 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:33.655335 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f885986c-44c9-419f-a368-0d2d79f9f088 None admin] PATH_INFO: `/v3/projects/61211724a3d74aaa873122b3d9d55a2e` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:33.656509 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f885986c-44c9-419f-a368-0d2d79f9f088 None admin] RBAC: Authorizing `identity:get_project(project_id=61211724a3d74aaa873122b3d9d55a2e)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:33.657934 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f885986c-44c9-419f-a368-0d2d79f9f088 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:33.659688 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 582/1160] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:21:33 2022] GET /identity/v3/projects/61211724a3d74aaa873122b3d9d55a2e => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:33.666635 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-429e615c-a1a1-4057-84a5-41dae11003ef None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:33.690672 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:33.766349 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-429e615c-a1a1-4057-84a5-41dae11003ef None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:33.800554 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aaaff84-1efc-4e38-a5e1-d9a68dcc454d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:33.801201 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aaaff84-1efc-4e38-a5e1-d9a68dcc454d None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:33.801489 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aaaff84-1efc-4e38-a5e1-d9a68dcc454d None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:33.801770 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aaaff84-1efc-4e38-a5e1-d9a68dcc454d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:33.802635 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aaaff84-1efc-4e38-a5e1-d9a68dcc454d None admin] RBAC: Adding query filter params (role.id=eb05bf286c894ea9bf44a6c023a4323b, scope.project.id=61211724a3d74aaa873122b3d9d55a2e, user.id=8c62e70fbb4d4134a70e21f79cf2ade0) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:21:33.802956 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aaaff84-1efc-4e38-a5e1-d9a68dcc454d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:33.805012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aaaff84-1efc-4e38-a5e1-d9a68dcc454d None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:33.814216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 579/1161] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:21:33 2022] GET /identity/v3/role_assignments?user.id=8c62e70fbb4d4134a70e21f79cf2ade0&scope.project.id=61211724a3d74aaa873122b3d9d55a2e&role.id=eb05bf286c894ea9bf44a6c023a4323b => generated 608 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:35.886018 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef648bc2-50cb-4870-a08e-66e93c14da50 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:35.886573 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef648bc2-50cb-4870-a08e-66e93c14da50 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:35.886573 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef648bc2-50cb-4870-a08e-66e93c14da50 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:35.887927 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 583/1162] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:21:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:21:35.893209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e569bc1-b8c7-48f5-9591-aa00d3b1f803 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:35.893647 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e569bc1-b8c7-48f5-9591-aa00d3b1f803 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:35.893783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e569bc1-b8c7-48f5-9591-aa00d3b1f803 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:35.910764 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-6e569bc1-b8c7-48f5-9591-aa00d3b1f803 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:21:35.911964 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-6e569bc1-b8c7-48f5-9591-aa00d3b1f803 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:21:35.979480 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 580/1163] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:21:35 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:21:36.001870 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef648bc2-50cb-4870-a08e-66e93c14da50 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:36.108757 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef648bc2-50cb-4870-a08e-66e93c14da50 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:36.111143 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:36.142395 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13d9262b-587e-415f-a948-ee5792c82aca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:36.143024 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d9262b-587e-415f-a948-ee5792c82aca None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:36.143314 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d9262b-587e-415f-a948-ee5792c82aca None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:36.143601 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d9262b-587e-415f-a948-ee5792c82aca None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:36.210311 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13d9262b-587e-415f-a948-ee5792c82aca None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:36.211895 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13d9262b-587e-415f-a948-ee5792c82aca None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:36.309170 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 584/1164] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:21:36 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 12 17:21:36.315200 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e569bc1-b8c7-48f5-9591-aa00d3b1f803 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:36.332693 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:36.412736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e569bc1-b8c7-48f5-9591-aa00d3b1f803 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:36.446231 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa112c4d-1c1f-46aa-bd4c-133600c6f7e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:36.447057 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa112c4d-1c1f-46aa-bd4c-133600c6f7e4 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:36.447345 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa112c4d-1c1f-46aa-bd4c-133600c6f7e4 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:36.447630 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa112c4d-1c1f-46aa-bd4c-133600c6f7e4 None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:36.454339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa112c4d-1c1f-46aa-bd4c-133600c6f7e4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:36.455781 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa112c4d-1c1f-46aa-bd4c-133600c6f7e4 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:36.470281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-fa112c4d-1c1f-46aa-bd4c-133600c6f7e4 None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Jul 12 17:21:36.470281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:21:36.470281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Jul 12 17:21:36.470281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Jul 12 17:21:36.470281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:21:36.470281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:21:36.470281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:21:36.470281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:21:36.470281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Jul 12 17:21:36.470281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:21:36.470281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 12 17:21:36.470281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:21:36.470281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:21:36.470281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:21:36.470281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:21:36.470281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:21:36.470281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:21:36.471804 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:21:36.471804 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:21:36.471804 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:21:36.471804 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:21:36.471804 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:21:36.471804 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:21:36.471804 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:21:36.471804 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:21:36.471804 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:21:36.471804 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:21:36.471804 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:21:36.471804 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:21:36.471804 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:21:36.471804 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:21:36.471804 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:21:36.471804 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:21:36.473109 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:21:36.473109 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:21:36.473109 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:21:36.473109 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:21:36.473109 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:21:36.473109 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:21:36.473109 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:21:36.473109 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:21:36.473109 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:21:36.473109 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:21:36.473109 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Jul 12 17:21:36.473109 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:21:36.473109 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Jul 12 17:21:36.473109 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:21:36.473109 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 581/1165] 10.222.0.16 () {56 vars in 1117 bytes} [Tue Jul 12 17:21:36 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:21:36.477886 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13d9262b-587e-415f-a948-ee5792c82aca None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:36.480956 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:36.575091 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13d9262b-587e-415f-a948-ee5792c82aca None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:36.610610 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b9862a-66c6-4d26-911e-61f67c445257 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:36.611279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b9862a-66c6-4d26-911e-61f67c445257 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:36.611566 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b9862a-66c6-4d26-911e-61f67c445257 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:36.611845 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b9862a-66c6-4d26-911e-61f67c445257 None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:36.612564 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b9862a-66c6-4d26-911e-61f67c445257 None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:21:36.612876 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b9862a-66c6-4d26-911e-61f67c445257 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:36.614234 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b9862a-66c6-4d26-911e-61f67c445257 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:36.621352 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 585/1166] 10.222.0.16 () {56 vars in 1115 bytes} [Tue Jul 12 17:21:36 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:38.658726 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c784a5-38b0-4833-8623-3b249210b476 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:38.659267 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c784a5-38b0-4833-8623-3b249210b476 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:38.659267 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c784a5-38b0-4833-8623-3b249210b476 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:38.660638 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 582/1167] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:21:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:21:38.666029 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b67e76-6000-46c8-ad96-d0bc426a8d42 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:38.666406 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b67e76-6000-46c8-ad96-d0bc426a8d42 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:38.666630 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b67e76-6000-46c8-ad96-d0bc426a8d42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:38.683379 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-f8b67e76-6000-46c8-ad96-d0bc426a8d42 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:21:38.684674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-f8b67e76-6000-46c8-ad96-d0bc426a8d42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:21:38.745466 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:38.751047 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 586/1168] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:21:38 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:21:38.774828 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c784a5-38b0-4833-8623-3b249210b476 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:38.853402 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:38.876369 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c784a5-38b0-4833-8623-3b249210b476 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:38.909933 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19aa08e-21cd-46a6-a8c8-3705cbbd1b27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:38.910622 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19aa08e-21cd-46a6-a8c8-3705cbbd1b27 None admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:38.910916 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19aa08e-21cd-46a6-a8c8-3705cbbd1b27 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:38.911200 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19aa08e-21cd-46a6-a8c8-3705cbbd1b27 None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:38.911949 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19aa08e-21cd-46a6-a8c8-3705cbbd1b27 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:38.913506 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19aa08e-21cd-46a6-a8c8-3705cbbd1b27 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:38.992951 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 583/1169] 10.222.0.16 () {60 vars in 1138 bytes} [Tue Jul 12 17:21:38 2022] POST /identity/v3/domains => generated 267 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:21:41.027133 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6a37ef-d792-415b-a4e4-d1a23da40e2e None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:41.027705 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6a37ef-d792-415b-a4e4-d1a23da40e2e None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:41.027705 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6a37ef-d792-415b-a4e4-d1a23da40e2e None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:41.029081 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 587/1170] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:21:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:21:41.034321 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72a69ac-d340-4a98-a3cd-507c3f4913b6 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:41.034856 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72a69ac-d340-4a98-a3cd-507c3f4913b6 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:41.034947 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72a69ac-d340-4a98-a3cd-507c3f4913b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:41.051708 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-f72a69ac-d340-4a98-a3cd-507c3f4913b6 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:21:41.052905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-f72a69ac-d340-4a98-a3cd-507c3f4913b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:21:41.121290 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 584/1171] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:21:41 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:21:41.152303 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6a37ef-d792-415b-a4e4-d1a23da40e2e None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:41.252629 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6a37ef-d792-415b-a4e4-d1a23da40e2e None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:41.285603 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f81b0e5-de9d-40e6-a349-794befe2c84c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:41.286379 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f81b0e5-de9d-40e6-a349-794befe2c84c None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:41.286691 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f81b0e5-de9d-40e6-a349-794befe2c84c None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:41.286966 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f81b0e5-de9d-40e6-a349-794befe2c84c None admin] PATH_INFO: `/v3/domains/ae52763a3b554e5087d61dafb05fd243` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:41.297137 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f81b0e5-de9d-40e6-a349-794befe2c84c None admin] RBAC: Authorizing `identity:get_domain(domain_id=ae52763a3b554e5087d61dafb05fd243)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:41.298606 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f81b0e5-de9d-40e6-a349-794befe2c84c None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:41.300225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 588/1172] 10.222.0.16 () {56 vars in 1183 bytes} [Tue Jul 12 17:21:41 2022] GET /identity/v3/domains/ae52763a3b554e5087d61dafb05fd243 => generated 267 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:41.306825 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72a69ac-d340-4a98-a3cd-507c3f4913b6 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:41.370448 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:41.403663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72a69ac-d340-4a98-a3cd-507c3f4913b6 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:41.436937 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ba17e0-a939-4e54-b613-fe2542dd5674 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:41.437597 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ba17e0-a939-4e54-b613-fe2542dd5674 None admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:41.437887 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ba17e0-a939-4e54-b613-fe2542dd5674 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:41.438170 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ba17e0-a939-4e54-b613-fe2542dd5674 None admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:41.438926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ba17e0-a939-4e54-b613-fe2542dd5674 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:41.440351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ba17e0-a939-4e54-b613-fe2542dd5674 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:41.556731 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 585/1173] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Jul 12 17:21:41 2022] POST /identity/v3/projects => generated 361 bytes in 250 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:21:43.641989 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ed881e-b05d-4a72-9aa5-e2591d7d38ae None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:43.642676 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ed881e-b05d-4a72-9aa5-e2591d7d38ae None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:43.642676 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ed881e-b05d-4a72-9aa5-e2591d7d38ae None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:43.644067 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 589/1174] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:21:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:21:43.649926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a2a789-e79d-4ef6-8dff-f58f7c3c7dae None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:43.650335 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a2a789-e79d-4ef6-8dff-f58f7c3c7dae None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:43.650532 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a2a789-e79d-4ef6-8dff-f58f7c3c7dae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:43.668294 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-21a2a789-e79d-4ef6-8dff-f58f7c3c7dae None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:21:43.669491 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-21a2a789-e79d-4ef6-8dff-f58f7c3c7dae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:21:43.735742 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 586/1175] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:21:43 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:21:43.760881 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ed881e-b05d-4a72-9aa5-e2591d7d38ae None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:43.819794 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:43.861191 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ed881e-b05d-4a72-9aa5-e2591d7d38ae None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:43.893819 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f05a61b-e4ee-4228-8803-70821463ede4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:43.894704 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f05a61b-e4ee-4228-8803-70821463ede4 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:43.894990 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f05a61b-e4ee-4228-8803-70821463ede4 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:43.895270 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f05a61b-e4ee-4228-8803-70821463ede4 None admin] PATH_INFO: `/v3/domains/ae52763a3b554e5087d61dafb05fd243` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:43.896438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f05a61b-e4ee-4228-8803-70821463ede4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=ae52763a3b554e5087d61dafb05fd243)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:43.898497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f05a61b-e4ee-4228-8803-70821463ede4 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:43.900116 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 590/1176] 10.222.0.16 () {56 vars in 1183 bytes} [Tue Jul 12 17:21:43 2022] GET /identity/v3/domains/ae52763a3b554e5087d61dafb05fd243 => generated 267 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:43.906097 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a2a789-e79d-4ef6-8dff-f58f7c3c7dae None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:43.953081 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:44.002350 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a2a789-e79d-4ef6-8dff-f58f7c3c7dae None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:44.037227 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cb0d45-f421-4ce8-8037-c5da5cadb5ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:44.037933 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cb0d45-f421-4ce8-8037-c5da5cadb5ec None admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:44.038223 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cb0d45-f421-4ce8-8037-c5da5cadb5ec None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:44.038537 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cb0d45-f421-4ce8-8037-c5da5cadb5ec None admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:44.039254 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38cb0d45-f421-4ce8-8037-c5da5cadb5ec None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:44.040687 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38cb0d45-f421-4ce8-8037-c5da5cadb5ec None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:44.143432 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 587/1177] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Jul 12 17:21:43 2022] POST /identity/v3/users => generated 311 bytes in 238 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:21:46.194316 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cdb4a3-3cb8-44bb-aa89-d0efb51067be None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:46.194908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cdb4a3-3cb8-44bb-aa89-d0efb51067be None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:46.194908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7cdb4a3-3cb8-44bb-aa89-d0efb51067be None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:46.196294 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 591/1178] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:21:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 12 17:21:46.201696 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af25a1d-1be2-499c-b6c3-188bbcfce2fc None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:46.202098 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af25a1d-1be2-499c-b6c3-188bbcfce2fc None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:46.202261 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af25a1d-1be2-499c-b6c3-188bbcfce2fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:46.218955 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-7af25a1d-1be2-499c-b6c3-188bbcfce2fc None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:21:46.220174 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-7af25a1d-1be2-499c-b6c3-188bbcfce2fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:21:46.288499 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 588/1179] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:21:46 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:21:46.312192 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7cdb4a3-3cb8-44bb-aa89-d0efb51067be None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:46.412761 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7cdb4a3-3cb8-44bb-aa89-d0efb51067be None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:46.438735 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:46.447685 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112d15d0-71f9-469b-a6e9-af32125c5f53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:46.448420 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112d15d0-71f9-469b-a6e9-af32125c5f53 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:46.448707 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112d15d0-71f9-469b-a6e9-af32125c5f53 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:46.448987 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112d15d0-71f9-469b-a6e9-af32125c5f53 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:46.459440 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112d15d0-71f9-469b-a6e9-af32125c5f53 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:21:46.467107 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-112d15d0-71f9-469b-a6e9-af32125c5f53 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:21:46.467107 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:21:46.467107 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:21:46.467107 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:21:46.467107 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:21:46.467107 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:21:46.467107 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:21:46.467107 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:21:46.467107 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:21:46.467107 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:21:46.467107 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:21:46.467107 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:21:46.467107 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:21:46.467107 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:21:46.467107 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:21:46.467107 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:21:46.468218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:21:46.468218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:21:46.468218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:21:46.468218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:21:46.468218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:21:46.468218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:21:46.468218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:21:46.468218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:21:46.468218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:21:46.468218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:21:46.468218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:21:46.468218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:21:46.468218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:21:46.468218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:21:46.468218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:21:46.468218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:21:46.469367 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:21:46.469367 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:21:46.469367 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:21:46.469367 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:21:46.469367 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:21:46.469367 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:21:46.469367 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:21:46.469367 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:21:46.469367 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:21:46.469367 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:21:46.469367 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:21:46.469367 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:21:46.469367 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 592/1180] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Jul 12 17:21:46 2022] GET /identity/v3/roles/admin => generated 83 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:21:46.474767 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af25a1d-1be2-499c-b6c3-188bbcfce2fc None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:46.508736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:46.574109 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af25a1d-1be2-499c-b6c3-188bbcfce2fc None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:46.608037 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a0207c-da67-4ad0-8a4a-4b03d5dbc4bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:46.608705 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a0207c-da67-4ad0-8a4a-4b03d5dbc4bf None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:46.608997 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a0207c-da67-4ad0-8a4a-4b03d5dbc4bf None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:46.609284 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a0207c-da67-4ad0-8a4a-4b03d5dbc4bf None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:46.610040 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7a0207c-da67-4ad0-8a4a-4b03d5dbc4bf None admin] RBAC: Adding query filter params (name=admin) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:21:46.610356 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7a0207c-da67-4ad0-8a4a-4b03d5dbc4bf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:46.611750 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7a0207c-da67-4ad0-8a4a-4b03d5dbc4bf None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:46.620603 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 589/1181] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:21:46 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:46.626442 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112d15d0-71f9-469b-a6e9-af32125c5f53 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:46.725032 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112d15d0-71f9-469b-a6e9-af32125c5f53 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:46.758994 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02f4401-ac55-4645-b0b7-5da01ea848c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:46.759861 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02f4401-ac55-4645-b0b7-5da01ea848c7 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:46.760143 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02f4401-ac55-4645-b0b7-5da01ea848c7 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:46.760421 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02f4401-ac55-4645-b0b7-5da01ea848c7 None admin] PATH_INFO: `/v3/users/71b37b506ac14541a6759a830cc24660` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:46.774147 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b02f4401-ac55-4645-b0b7-5da01ea848c7 None admin] RBAC: Authorizing `identity:get_user(user_id=71b37b506ac14541a6759a830cc24660)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:46.775615 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b02f4401-ac55-4645-b0b7-5da01ea848c7 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:46.777264 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 593/1182] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:21:46 2022] GET /identity/v3/users/71b37b506ac14541a6759a830cc24660 => generated 311 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:46.783263 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a0207c-da67-4ad0-8a4a-4b03d5dbc4bf None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:46.877415 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:46.883230 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a0207c-da67-4ad0-8a4a-4b03d5dbc4bf None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:46.917725 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7abd04-7dc1-47d9-9a14-dcf74e5e312e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:46.918373 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7abd04-7dc1-47d9-9a14-dcf74e5e312e None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:46.918732 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7abd04-7dc1-47d9-9a14-dcf74e5e312e None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:46.919017 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7abd04-7dc1-47d9-9a14-dcf74e5e312e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:46.986099 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7abd04-7dc1-47d9-9a14-dcf74e5e312e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:46.987499 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7abd04-7dc1-47d9-9a14-dcf74e5e312e None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:47.080538 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:47.087268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 590/1183] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:21:46 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 12 17:21:47.093397 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02f4401-ac55-4645-b0b7-5da01ea848c7 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:47.140117 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:47.193967 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02f4401-ac55-4645-b0b7-5da01ea848c7 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:47.228361 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58bf3da-c1ef-4b24-944d-3efbb53d8616 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:47.229225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58bf3da-c1ef-4b24-944d-3efbb53d8616 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:47.229513 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58bf3da-c1ef-4b24-944d-3efbb53d8616 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:47.229792 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58bf3da-c1ef-4b24-944d-3efbb53d8616 None admin] PATH_INFO: `/v3/projects/e8112c34006f4265b592f1ac20cb4290` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:47.231006 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58bf3da-c1ef-4b24-944d-3efbb53d8616 None admin] RBAC: Authorizing `identity:get_project(project_id=e8112c34006f4265b592f1ac20cb4290)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:47.232441 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58bf3da-c1ef-4b24-944d-3efbb53d8616 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:47.234149 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 594/1184] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:21:47 2022] GET /identity/v3/projects/e8112c34006f4265b592f1ac20cb4290 => generated 361 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:47.240696 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7abd04-7dc1-47d9-9a14-dcf74e5e312e None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:47.338846 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7abd04-7dc1-47d9-9a14-dcf74e5e312e None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:47.372086 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd142adc-e973-4f7b-a58b-5d5aedbbbcae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:47.372722 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd142adc-e973-4f7b-a58b-5d5aedbbbcae None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:47.373009 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd142adc-e973-4f7b-a58b-5d5aedbbbcae None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:47.373288 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd142adc-e973-4f7b-a58b-5d5aedbbbcae None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:47.374112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd142adc-e973-4f7b-a58b-5d5aedbbbcae None admin] RBAC: Adding query filter params (role.id=eb05bf286c894ea9bf44a6c023a4323b, scope.project.id=e8112c34006f4265b592f1ac20cb4290, user.id=71b37b506ac14541a6759a830cc24660) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:21:47.374456 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd142adc-e973-4f7b-a58b-5d5aedbbbcae None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:47.375883 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd142adc-e973-4f7b-a58b-5d5aedbbbcae None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:47.381343 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 591/1185] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:21:47 2022] GET /identity/v3/role_assignments?user.id=71b37b506ac14541a6759a830cc24660&scope.project.id=e8112c34006f4265b592f1ac20cb4290&role.id=eb05bf286c894ea9bf44a6c023a4323b => generated 259 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:49.408078 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e804f5b-27c7-47cd-b897-3f6f1b114226 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:49.408632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e804f5b-27c7-47cd-b897-3f6f1b114226 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:49.408632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e804f5b-27c7-47cd-b897-3f6f1b114226 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:49.410024 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 595/1186] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:21:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:21:49.415845 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318279de-6817-4b07-ab29-ad7c6c23cde9 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:49.416276 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318279de-6817-4b07-ab29-ad7c6c23cde9 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:49.416400 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318279de-6817-4b07-ab29-ad7c6c23cde9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:49.433389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-318279de-6817-4b07-ab29-ad7c6c23cde9 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:21:49.434715 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-318279de-6817-4b07-ab29-ad7c6c23cde9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:21:49.491238 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:49.502712 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 592/1187] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:21:49 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:21:49.526044 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e804f5b-27c7-47cd-b897-3f6f1b114226 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:49.627284 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e804f5b-27c7-47cd-b897-3f6f1b114226 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:49.640681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:49.661471 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01221baf-3e53-4c9e-a1a4-c979cc3e892c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:49.662238 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01221baf-3e53-4c9e-a1a4-c979cc3e892c None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:49.663049 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01221baf-3e53-4c9e-a1a4-c979cc3e892c None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:49.663329 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01221baf-3e53-4c9e-a1a4-c979cc3e892c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:49.673803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01221baf-3e53-4c9e-a1a4-c979cc3e892c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:21:49.681497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-01221baf-3e53-4c9e-a1a4-c979cc3e892c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:21:49.681497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:21:49.681497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:21:49.681497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:21:49.681497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:21:49.681497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:21:49.681497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:21:49.681497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:21:49.681497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:21:49.681497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:21:49.681497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:21:49.681497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:21:49.681497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:21:49.681497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:21:49.681497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:21:49.681497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:21:49.682708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:21:49.682708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:21:49.682708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:21:49.682708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:21:49.682708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:21:49.682708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:21:49.682708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:21:49.682708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:21:49.682708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:21:49.682708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:21:49.682708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:21:49.682708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:21:49.682708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:21:49.682708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:21:49.682708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:21:49.682708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:21:49.683827 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:21:49.683827 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:21:49.683827 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:21:49.683827 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:21:49.683827 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:21:49.683827 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:21:49.683827 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:21:49.683827 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:21:49.683827 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:21:49.683827 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:21:49.683827 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:21:49.683827 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:21:49.683827 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 596/1188] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Jul 12 17:21:49 2022] GET /identity/v3/roles/admin => generated 83 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:21:49.689027 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318279de-6817-4b07-ab29-ad7c6c23cde9 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:49.789574 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318279de-6817-4b07-ab29-ad7c6c23cde9 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:49.822897 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0949fbaa-8314-44f2-9ee1-7c21ce84fbc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:49.823565 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0949fbaa-8314-44f2-9ee1-7c21ce84fbc3 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:49.823858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0949fbaa-8314-44f2-9ee1-7c21ce84fbc3 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:49.824144 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0949fbaa-8314-44f2-9ee1-7c21ce84fbc3 None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:49.824891 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0949fbaa-8314-44f2-9ee1-7c21ce84fbc3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:21:49.825206 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0949fbaa-8314-44f2-9ee1-7c21ce84fbc3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:49.826592 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0949fbaa-8314-44f2-9ee1-7c21ce84fbc3 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:49.835312 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 593/1189] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:21:49 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:21:49.841258 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01221baf-3e53-4c9e-a1a4-c979cc3e892c None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:49.938720 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01221baf-3e53-4c9e-a1a4-c979cc3e892c None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:49.972146 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc17b00-3434-450f-aedc-f8a4e7e3820a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:49.973014 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc17b00-3434-450f-aedc-f8a4e7e3820a None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:49.973294 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc17b00-3434-450f-aedc-f8a4e7e3820a None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:49.973572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc17b00-3434-450f-aedc-f8a4e7e3820a None admin] PATH_INFO: `/v3/users/71b37b506ac14541a6759a830cc24660` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:49.974818 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc17b00-3434-450f-aedc-f8a4e7e3820a None admin] RBAC: Authorizing `identity:get_user(user_id=71b37b506ac14541a6759a830cc24660)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:49.976232 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc17b00-3434-450f-aedc-f8a4e7e3820a None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:49.977857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 597/1190] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:21:49 2022] GET /identity/v3/users/71b37b506ac14541a6759a830cc24660 => generated 311 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:49.983888 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0949fbaa-8314-44f2-9ee1-7c21ce84fbc3 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:50.036905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:50.084652 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0949fbaa-8314-44f2-9ee1-7c21ce84fbc3 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:50.119401 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10a07a4-5257-4d9c-928c-8593a6c986b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:50.120258 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10a07a4-5257-4d9c-928c-8593a6c986b7 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:50.120546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10a07a4-5257-4d9c-928c-8593a6c986b7 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:50.120825 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10a07a4-5257-4d9c-928c-8593a6c986b7 None admin] PATH_INFO: `/v3/projects/e8112c34006f4265b592f1ac20cb4290` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:50.122004 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10a07a4-5257-4d9c-928c-8593a6c986b7 None admin] RBAC: Authorizing `identity:get_project(project_id=e8112c34006f4265b592f1ac20cb4290)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:50.123463 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10a07a4-5257-4d9c-928c-8593a6c986b7 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:50.125162 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 594/1191] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:21:49 2022] GET /identity/v3/projects/e8112c34006f4265b592f1ac20cb4290 => generated 361 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:50.131842 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc17b00-3434-450f-aedc-f8a4e7e3820a None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:50.173058 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:50.231102 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc17b00-3434-450f-aedc-f8a4e7e3820a None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:50.263792 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa6fc31-9c40-46ff-a9f1-2b5a1aee832c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:50.264669 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa6fc31-9c40-46ff-a9f1-2b5a1aee832c None admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:50.264952 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa6fc31-9c40-46ff-a9f1-2b5a1aee832c None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:50.265227 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa6fc31-9c40-46ff-a9f1-2b5a1aee832c None admin] PATH_INFO: `/v3/projects/e8112c34006f4265b592f1ac20cb4290/users/71b37b506ac14541a6759a830cc24660/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:50.267177 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa6fc31-9c40-46ff-a9f1-2b5a1aee832c None admin] RBAC: Authorizing `identity:create_grant(project_id=e8112c34006f4265b592f1ac20cb4290, user_id=71b37b506ac14541a6759a830cc24660, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:50.268599 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa6fc31-9c40-46ff-a9f1-2b5a1aee832c None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:50.354697 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 598/1192] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Jul 12 17:21:50 2022] PUT /identity/v3/projects/e8112c34006f4265b592f1ac20cb4290/users/71b37b506ac14541a6759a830cc24660/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:21:52.362341 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aecd92bc-d666-4ede-9092-3ed0fa6bada3 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:52.362895 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aecd92bc-d666-4ede-9092-3ed0fa6bada3 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:52.362989 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aecd92bc-d666-4ede-9092-3ed0fa6bada3 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:52.364366 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 595/1193] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:21:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:21:52.369764 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f87314a-7302-4c55-80d0-598836c59b0d None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:52.370204 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f87314a-7302-4c55-80d0-598836c59b0d None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:52.370335 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f87314a-7302-4c55-80d0-598836c59b0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:52.387870 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-8f87314a-7302-4c55-80d0-598836c59b0d None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:21:52.389071 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-8f87314a-7302-4c55-80d0-598836c59b0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:21:52.455069 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 599/1194] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:21:52 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:21:52.479132 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aecd92bc-d666-4ede-9092-3ed0fa6bada3 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:52.581610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aecd92bc-d666-4ede-9092-3ed0fa6bada3 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:52.603115 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:52.617474 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d78dc4-3d47-4fcf-9d87-e6082f5880b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:52.617880 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d78dc4-3d47-4fcf-9d87-e6082f5880b9 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:52.618170 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d78dc4-3d47-4fcf-9d87-e6082f5880b9 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:52.618476 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d78dc4-3d47-4fcf-9d87-e6082f5880b9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:52.628553 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d78dc4-3d47-4fcf-9d87-e6082f5880b9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:21:52.637151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-c2d78dc4-3d47-4fcf-9d87-e6082f5880b9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:21:52.637151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:21:52.637151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:21:52.637151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:21:52.637151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:21:52.637151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:21:52.637151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:21:52.637151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:21:52.637151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:21:52.637151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:21:52.637151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:21:52.637151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:21:52.637151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:21:52.637151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:21:52.637151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:21:52.637151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:21:52.638295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:21:52.638295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:21:52.638295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:21:52.638295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:21:52.638295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:21:52.638295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:21:52.638295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:21:52.638295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:21:52.638295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:21:52.638295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:21:52.638295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:21:52.638295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:21:52.638295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:21:52.638295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:21:52.638295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:21:52.638295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:21:52.639379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:21:52.639379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:21:52.639379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:21:52.639379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:21:52.639379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:21:52.639379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:21:52.639379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:21:52.639379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:21:52.639379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:21:52.639379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:21:52.639379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:21:52.639379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:21:52.639379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 596/1195] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Jul 12 17:21:52 2022] GET /identity/v3/roles/admin => generated 83 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:21:52.644746 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f87314a-7302-4c55-80d0-598836c59b0d None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:52.674187 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:52.744091 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f87314a-7302-4c55-80d0-598836c59b0d None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:52.778040 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b548530e-c67f-4cd7-b8d8-30e58450bda2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:52.778722 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b548530e-c67f-4cd7-b8d8-30e58450bda2 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:52.779007 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b548530e-c67f-4cd7-b8d8-30e58450bda2 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:52.779279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b548530e-c67f-4cd7-b8d8-30e58450bda2 None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:52.780046 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b548530e-c67f-4cd7-b8d8-30e58450bda2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:21:52.780352 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b548530e-c67f-4cd7-b8d8-30e58450bda2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:52.781702 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b548530e-c67f-4cd7-b8d8-30e58450bda2 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:52.790378 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 600/1196] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:21:52 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:52.796246 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d78dc4-3d47-4fcf-9d87-e6082f5880b9 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:52.895589 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d78dc4-3d47-4fcf-9d87-e6082f5880b9 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:52.928950 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899482c3-14e4-4e7f-824b-bc516fb07c47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:52.929801 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899482c3-14e4-4e7f-824b-bc516fb07c47 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:52.930089 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899482c3-14e4-4e7f-824b-bc516fb07c47 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:52.930369 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899482c3-14e4-4e7f-824b-bc516fb07c47 None admin] PATH_INFO: `/v3/users/71b37b506ac14541a6759a830cc24660` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:52.931585 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-899482c3-14e4-4e7f-824b-bc516fb07c47 None admin] RBAC: Authorizing `identity:get_user(user_id=71b37b506ac14541a6759a830cc24660)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:52.933016 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-899482c3-14e4-4e7f-824b-bc516fb07c47 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:52.934681 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 597/1197] 10.222.0.16 () {56 vars in 1177 bytes} [Tue Jul 12 17:21:52 2022] GET /identity/v3/users/71b37b506ac14541a6759a830cc24660 => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:52.940428 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b548530e-c67f-4cd7-b8d8-30e58450bda2 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:53.030517 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:53.039431 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b548530e-c67f-4cd7-b8d8-30e58450bda2 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:53.072412 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc462355-e406-4ed4-bd59-6c17c349e9f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:53.073060 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc462355-e406-4ed4-bd59-6c17c349e9f5 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:53.073350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc462355-e406-4ed4-bd59-6c17c349e9f5 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:53.073635 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc462355-e406-4ed4-bd59-6c17c349e9f5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:53.140321 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc462355-e406-4ed4-bd59-6c17c349e9f5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:53.141766 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc462355-e406-4ed4-bd59-6c17c349e9f5 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:53.233842 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:53.243239 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 601/1198] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:21:52 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 12 17:21:53.249218 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899482c3-14e4-4e7f-824b-bc516fb07c47 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:53.299935 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:21:53.351591 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899482c3-14e4-4e7f-824b-bc516fb07c47 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:53.384389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd588de-e31a-41cd-82b0-3b0432031266 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:53.385244 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd588de-e31a-41cd-82b0-3b0432031266 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:53.385532 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd588de-e31a-41cd-82b0-3b0432031266 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:53.385814 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd588de-e31a-41cd-82b0-3b0432031266 None admin] PATH_INFO: `/v3/projects/e8112c34006f4265b592f1ac20cb4290` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:53.387030 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd588de-e31a-41cd-82b0-3b0432031266 None admin] RBAC: Authorizing `identity:get_project(project_id=e8112c34006f4265b592f1ac20cb4290)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:53.388453 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd588de-e31a-41cd-82b0-3b0432031266 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:53.390148 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 598/1199] 10.222.0.16 () {56 vars in 1186 bytes} [Tue Jul 12 17:21:53 2022] GET /identity/v3/projects/e8112c34006f4265b592f1ac20cb4290 => generated 361 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:21:53.397088 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc462355-e406-4ed4-bd59-6c17c349e9f5 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:21:53.495400 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc462355-e406-4ed4-bd59-6c17c349e9f5 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:21:53.529915 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83af8e77-cb11-4851-814e-0a2853ea0545 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:21:53.530597 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83af8e77-cb11-4851-814e-0a2853ea0545 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:21:53.530889 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83af8e77-cb11-4851-814e-0a2853ea0545 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:21:53.531175 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83af8e77-cb11-4851-814e-0a2853ea0545 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:21:53.532010 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83af8e77-cb11-4851-814e-0a2853ea0545 None admin] RBAC: Adding query filter params (role.id=eb05bf286c894ea9bf44a6c023a4323b, scope.project.id=e8112c34006f4265b592f1ac20cb4290, user.id=71b37b506ac14541a6759a830cc24660) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:21:53.532327 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83af8e77-cb11-4851-814e-0a2853ea0545 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:21:53.533965 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83af8e77-cb11-4851-814e-0a2853ea0545 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:21:53.540561 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 602/1200] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:21:53 2022] GET /identity/v3/role_assignments?user.id=71b37b506ac14541a6759a830cc24660&scope.project.id=e8112c34006f4265b592f1ac20cb4290&role.id=eb05bf286c894ea9bf44a6c023a4323b => generated 608 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:22:03.380066 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=42 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:22:03.537523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=72 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:11.290131 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a568ac-d272-47ea-a09e-452c059c6341 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:11.290759 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a568ac-d272-47ea-a09e-452c059c6341 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:11.290759 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a568ac-d272-47ea-a09e-452c059c6341 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:11.292091 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 599/1201] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:23:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:23:11.298014 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e649f93-7181-4950-a424-541005c2e5cd None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:11.298592 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e649f93-7181-4950-a424-541005c2e5cd None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:11.298592 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e649f93-7181-4950-a424-541005c2e5cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:11.316284 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-8e649f93-7181-4950-a424-541005c2e5cd None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:23:11.317505 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-8e649f93-7181-4950-a424-541005c2e5cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:23:11.385247 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 603/1202] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:23:11 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:23:11.411119 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91a568ac-d272-47ea-a09e-452c059c6341 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:11.520289 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91a568ac-d272-47ea-a09e-452c059c6341 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:11.555476 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a33d05d-790e-4932-9fab-e888244eefe7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:11.556205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a33d05d-790e-4932-9fab-e888244eefe7 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:11.556496 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a33d05d-790e-4932-9fab-e888244eefe7 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:11.556776 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a33d05d-790e-4932-9fab-e888244eefe7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:11.567339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a33d05d-790e-4932-9fab-e888244eefe7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:11.568846 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a33d05d-790e-4932-9fab-e888244eefe7 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:11.587216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-8a33d05d-790e-4932-9fab-e888244eefe7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:23:11.587216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:11.587216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:23:11.587216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:23:11.587216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:23:11.587216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:23:11.587216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:23:11.587216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:23:11.587216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:23:11.587216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:23:11.587216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:23:11.587216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:23:11.587216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:23:11.587216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:23:11.587216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:23:11.593076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:23:11.593076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:23:11.593076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:23:11.593076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:23:11.593076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:23:11.593076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:23:11.593076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:23:11.593076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:23:11.593076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:23:11.593076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:23:11.593076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:23:11.593076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:23:11.593076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:23:11.593076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:23:11.593076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:23:11.593076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:23:11.593076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:23:11.594614 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:23:11.594614 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:23:11.594614 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:23:11.594614 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:23:11.594614 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 600/1203] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:23:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:23:11.595582 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e649f93-7181-4950-a424-541005c2e5cd None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:11.697196 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e649f93-7181-4950-a424-541005c2e5cd None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:11.726203 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:11.732055 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308e3484-4139-40fa-bd11-1d23cf43ebd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:11.732742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308e3484-4139-40fa-bd11-1d23cf43ebd5 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:11.733031 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308e3484-4139-40fa-bd11-1d23cf43ebd5 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:11.733314 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308e3484-4139-40fa-bd11-1d23cf43ebd5 None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:11.734125 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-308e3484-4139-40fa-bd11-1d23cf43ebd5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:11.734501 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-308e3484-4139-40fa-bd11-1d23cf43ebd5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:11.736494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-308e3484-4139-40fa-bd11-1d23cf43ebd5 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:11.747836 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 604/1204] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:23:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:23:11.753797 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a33d05d-790e-4932-9fab-e888244eefe7 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:11.807621 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:11.854591 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a33d05d-790e-4932-9fab-e888244eefe7 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:11.887360 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bac9b31-d7c7-49f8-a7f3-70457e0ee6cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:11.888027 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bac9b31-d7c7-49f8-a7f3-70457e0ee6cc None admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:11.888336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bac9b31-d7c7-49f8-a7f3-70457e0ee6cc None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:11.888595 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bac9b31-d7c7-49f8-a7f3-70457e0ee6cc None admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:11.889306 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bac9b31-d7c7-49f8-a7f3-70457e0ee6cc None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:11.891287 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bac9b31-d7c7-49f8-a7f3-70457e0ee6cc None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:12.003203 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 601/1205] 10.222.0.16 () {60 vars in 1132 bytes} [Tue Jul 12 17:23:11 2022] POST /identity/v3/users => generated 251 bytes in 250 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:23:14.066797 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c5d7f0-1aee-449b-804e-59b0dca4c151 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:14.067350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c5d7f0-1aee-449b-804e-59b0dca4c151 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:14.067350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c5d7f0-1aee-449b-804e-59b0dca4c151 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:14.068744 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 605/1206] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:23:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:23:14.073862 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9eeea99-1f26-462d-a381-19bbc9791915 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:14.074291 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9eeea99-1f26-462d-a381-19bbc9791915 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:14.074450 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9eeea99-1f26-462d-a381-19bbc9791915 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:14.092045 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-b9eeea99-1f26-462d-a381-19bbc9791915 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:23:14.093273 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-b9eeea99-1f26-462d-a381-19bbc9791915 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:23:14.168800 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 602/1207] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:23:14 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:23:14.193320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c5d7f0-1aee-449b-804e-59b0dca4c151 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:14.299559 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c5d7f0-1aee-449b-804e-59b0dca4c151 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:14.333591 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a98672ff-e8c6-4118-9ca4-9ff9325fd60c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:14.334348 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98672ff-e8c6-4118-9ca4-9ff9325fd60c None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:14.334659 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98672ff-e8c6-4118-9ca4-9ff9325fd60c None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:14.334935 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98672ff-e8c6-4118-9ca4-9ff9325fd60c None admin] PATH_INFO: `/v3/roles/service` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:14.346157 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a98672ff-e8c6-4118-9ca4-9ff9325fd60c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:23:14.353736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-a98672ff-e8c6-4118-9ca4-9ff9325fd60c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:23:14.353736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:14.353736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:23:14.353736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:23:14.353736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:23:14.353736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:23:14.353736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:23:14.353736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:23:14.353736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:23:14.353736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:23:14.353736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:23:14.353736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:23:14.353736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:23:14.353736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:23:14.353736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:23:14.353736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:23:14.354935 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:23:14.354935 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:23:14.354935 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:23:14.354935 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:23:14.354935 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:23:14.354935 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:23:14.354935 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:23:14.354935 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:23:14.354935 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:23:14.354935 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:23:14.354935 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:23:14.354935 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:23:14.354935 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:23:14.354935 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:23:14.354935 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:23:14.354935 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:23:14.356201 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:23:14.356201 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:23:14.356201 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:23:14.356201 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:23:14.356201 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:23:14.356201 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:23:14.356201 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:23:14.356201 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:23:14.356201 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:23:14.356201 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:23:14.356201 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:23:14.356201 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:23:14.356201 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 606/1208] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Jul 12 17:23:14 2022] GET /identity/v3/roles/service => generated 85 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:23:14.361729 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9eeea99-1f26-462d-a381-19bbc9791915 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:14.401933 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:14.460936 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9eeea99-1f26-462d-a381-19bbc9791915 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:14.496603 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f36b59-c89f-424c-bc23-60df62c59c3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:14.497275 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f36b59-c89f-424c-bc23-60df62c59c3a None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:14.497565 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f36b59-c89f-424c-bc23-60df62c59c3a None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:14.497850 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f36b59-c89f-424c-bc23-60df62c59c3a None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:14.498626 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f36b59-c89f-424c-bc23-60df62c59c3a None admin] RBAC: Adding query filter params (name=service) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:14.498944 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f36b59-c89f-424c-bc23-60df62c59c3a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:14.500350 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f36b59-c89f-424c-bc23-60df62c59c3a None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:14.508275 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 603/1209] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Jul 12 17:23:14 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:23:14.514126 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a98672ff-e8c6-4118-9ca4-9ff9325fd60c None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:14.565832 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:14.616361 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a98672ff-e8c6-4118-9ca4-9ff9325fd60c None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:14.650704 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e4bbe5-d984-4fd7-bd8c-5f7fb3c827d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:14.651485 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e4bbe5-d984-4fd7-bd8c-5f7fb3c827d0 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:14.651770 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e4bbe5-d984-4fd7-bd8c-5f7fb3c827d0 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:14.652043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e4bbe5-d984-4fd7-bd8c-5f7fb3c827d0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:14.663053 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e4bbe5-d984-4fd7-bd8c-5f7fb3c827d0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:14.664525 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e4bbe5-d984-4fd7-bd8c-5f7fb3c827d0 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:14.682284 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-90e4bbe5-d984-4fd7-bd8c-5f7fb3c827d0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:23:14.682284 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:14.682284 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:23:14.682284 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:23:14.682284 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:23:14.682284 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:23:14.682284 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:23:14.682284 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:23:14.682284 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:23:14.682284 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:23:14.682284 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:23:14.682284 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:23:14.682284 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:23:14.682284 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:23:14.682284 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:23:14.683384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:23:14.683384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:23:14.683384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:23:14.683384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:23:14.683384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:23:14.683384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:23:14.683384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:23:14.683384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:23:14.683384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:23:14.683384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:23:14.683384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:23:14.683384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:23:14.683384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:23:14.683384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:23:14.683384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:23:14.683384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:23:14.683384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:23:14.684504 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:23:14.684504 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:23:14.684504 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:23:14.684504 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:23:14.684504 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 607/1210] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:23:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:23:14.689719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f36b59-c89f-424c-bc23-60df62c59c3a None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:14.786544 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:14.789414 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f36b59-c89f-424c-bc23-60df62c59c3a None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:14.823469 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a920e155-443c-4c8f-ae2e-b8c34f472170 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:14.824148 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a920e155-443c-4c8f-ae2e-b8c34f472170 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:14.824437 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a920e155-443c-4c8f-ae2e-b8c34f472170 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:14.824721 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a920e155-443c-4c8f-ae2e-b8c34f472170 None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:14.825546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a920e155-443c-4c8f-ae2e-b8c34f472170 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:14.825795 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a920e155-443c-4c8f-ae2e-b8c34f472170 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:14.827193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a920e155-443c-4c8f-ae2e-b8c34f472170 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:14.839686 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 604/1211] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:23:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:23:14.846083 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e4bbe5-d984-4fd7-bd8c-5f7fb3c827d0 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:14.945386 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e4bbe5-d984-4fd7-bd8c-5f7fb3c827d0 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:14.946873 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:14.979864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69c2a1a-6595-4ee9-9978-6389cf5d8b5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:14.980665 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69c2a1a-6595-4ee9-9978-6389cf5d8b5f None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:14.980951 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69c2a1a-6595-4ee9-9978-6389cf5d8b5f None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:14.981229 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69c2a1a-6595-4ee9-9978-6389cf5d8b5f None admin] PATH_INFO: `/v3/users/placement` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:14.988934 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e69c2a1a-6595-4ee9-9978-6389cf5d8b5f None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:14.990373 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e69c2a1a-6595-4ee9-9978-6389cf5d8b5f None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:15.004475 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-e69c2a1a-6595-4ee9-9978-6389cf5d8b5f None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 12 17:23:15.004475 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:15.004475 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:23:15.004475 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:23:15.004475 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:23:15.004475 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:23:15.004475 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:23:15.004475 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:23:15.004475 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:23:15.004475 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:23:15.004475 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:23:15.004475 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:23:15.004475 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:23:15.004475 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:23:15.004475 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:23:15.004475 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:23:15.005674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:23:15.005674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:23:15.005674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:23:15.005674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:23:15.005674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:23:15.005674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:23:15.005674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:23:15.005674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:23:15.005674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:23:15.005674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:23:15.005674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:23:15.005674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:23:15.005674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:23:15.005674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:23:15.005674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:23:15.005674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:23:15.005674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:23:15.005674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:23:15.006904 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:23:15.006904 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:23:15.006904 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:23:15.006904 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:23:15.006904 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:23:15.006904 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:23:15.006904 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:23:15.006904 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:23:15.006904 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 12 17:23:15.006904 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:23:15.006904 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 608/1212] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:23:14 2022] GET /identity/v3/users/placement => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:23:15.011543 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a920e155-443c-4c8f-ae2e-b8c34f472170 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:15.114063 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a920e155-443c-4c8f-ae2e-b8c34f472170 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:15.147074 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c60e973-6731-421f-b515-c1a54f160944 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:15.147745 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c60e973-6731-421f-b515-c1a54f160944 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:15.148037 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c60e973-6731-421f-b515-c1a54f160944 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:15.148321 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c60e973-6731-421f-b515-c1a54f160944 None admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:15.149156 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c60e973-6731-421f-b515-c1a54f160944 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:15.149472 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c60e973-6731-421f-b515-c1a54f160944 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:15.150944 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c60e973-6731-421f-b515-c1a54f160944 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:15.168479 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 605/1213] 10.222.0.16 () {56 vars in 1143 bytes} [Tue Jul 12 17:23:15 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:23:15.174345 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69c2a1a-6595-4ee9-9978-6389cf5d8b5f None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:15.276789 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69c2a1a-6595-4ee9-9978-6389cf5d8b5f None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:15.311688 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c8b51e-f7e6-490e-bafa-e2b1c97bd96f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:15.312886 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c8b51e-f7e6-490e-bafa-e2b1c97bd96f None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:15.313239 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c8b51e-f7e6-490e-bafa-e2b1c97bd96f None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:15.313713 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c8b51e-f7e6-490e-bafa-e2b1c97bd96f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:15.344470 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:15.382127 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c8b51e-f7e6-490e-bafa-e2b1c97bd96f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:15.383779 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c8b51e-f7e6-490e-bafa-e2b1c97bd96f None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:15.482389 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 609/1214] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:23:15 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 12 17:23:15.488463 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c60e973-6731-421f-b515-c1a54f160944 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:15.508174 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:15.591033 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c60e973-6731-421f-b515-c1a54f160944 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:15.628585 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb0c947-f0bb-4143-842a-dfc172c0577d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:15.629513 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb0c947-f0bb-4143-842a-dfc172c0577d None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:15.629819 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb0c947-f0bb-4143-842a-dfc172c0577d None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:15.630104 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb0c947-f0bb-4143-842a-dfc172c0577d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:15.640951 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb0c947-f0bb-4143-842a-dfc172c0577d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:15.656952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb0c947-f0bb-4143-842a-dfc172c0577d None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:15.660229 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-acb0c947-f0bb-4143-842a-dfc172c0577d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:23:15.660229 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:15.660229 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:23:15.660229 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:23:15.660229 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:23:15.660229 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:23:15.660229 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:23:15.660229 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:23:15.660229 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:23:15.660229 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:23:15.660229 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:23:15.660229 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:23:15.660229 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:23:15.660229 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:23:15.660229 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:23:15.661312 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:23:15.661312 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:23:15.661312 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:23:15.661312 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:23:15.661312 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:23:15.661312 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:23:15.661312 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:23:15.661312 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:23:15.661312 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:23:15.661312 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:23:15.661312 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:23:15.661312 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:23:15.661312 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:23:15.661312 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:23:15.661312 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:23:15.661312 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:23:15.661312 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:23:15.662770 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:23:15.662770 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:23:15.662770 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:23:15.662770 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:23:15.662770 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 606/1215] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:23:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:23:15.667425 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c8b51e-f7e6-490e-bafa-e2b1c97bd96f None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:15.728569 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:15.768209 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c8b51e-f7e6-490e-bafa-e2b1c97bd96f None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:15.802527 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c295a6e2-bb35-4430-8a31-adcca19458b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:15.803197 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c295a6e2-bb35-4430-8a31-adcca19458b6 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:15.803486 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c295a6e2-bb35-4430-8a31-adcca19458b6 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:15.803765 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c295a6e2-bb35-4430-8a31-adcca19458b6 None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:15.804524 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c295a6e2-bb35-4430-8a31-adcca19458b6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:15.804808 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c295a6e2-bb35-4430-8a31-adcca19458b6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:15.806231 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c295a6e2-bb35-4430-8a31-adcca19458b6 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:15.820141 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 610/1216] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:23:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:23:15.825712 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb0c947-f0bb-4143-842a-dfc172c0577d None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:15.897233 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:15.926653 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb0c947-f0bb-4143-842a-dfc172c0577d None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:15.960436 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5e3b5a-acff-44e9-8383-e7cc8185c506 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:15.961211 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5e3b5a-acff-44e9-8383-e7cc8185c506 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:15.961494 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5e3b5a-acff-44e9-8383-e7cc8185c506 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:15.961768 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5e3b5a-acff-44e9-8383-e7cc8185c506 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:15.969033 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca5e3b5a-acff-44e9-8383-e7cc8185c506 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:15.970468 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca5e3b5a-acff-44e9-8383-e7cc8185c506 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:15.983772 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-ca5e3b5a-acff-44e9-8383-e7cc8185c506 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:23:15.983772 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:15.983772 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:23:15.983772 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:23:15.983772 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:23:15.983772 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:23:15.983772 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:23:15.983772 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:23:15.983772 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:23:15.983772 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:23:15.983772 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:23:15.983772 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:23:15.983772 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:23:15.983772 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:23:15.983772 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:23:15.987214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:23:15.987214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:23:15.987214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:23:15.987214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:23:15.987214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:23:15.987214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:23:15.987214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:23:15.987214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:23:15.987214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:23:15.987214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:23:15.987214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:23:15.987214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:23:15.987214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:23:15.987214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:23:15.987214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:23:15.987214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:23:15.987214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:23:15.988129 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:23:15.988129 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:23:15.988129 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:23:15.988129 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:23:15.988129 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:23:15.988129 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:23:15.988129 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:23:15.988129 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:23:15.988129 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 607/1217] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:23:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:23:15.991446 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c295a6e2-bb35-4430-8a31-adcca19458b6 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:16.091930 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c295a6e2-bb35-4430-8a31-adcca19458b6 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:16.114532 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:16.125519 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-625731b9-5898-44fc-9492-2bf62fe7b401 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:16.126184 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625731b9-5898-44fc-9492-2bf62fe7b401 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:16.126492 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625731b9-5898-44fc-9492-2bf62fe7b401 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:16.126772 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-625731b9-5898-44fc-9492-2bf62fe7b401 None admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:16.127556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625731b9-5898-44fc-9492-2bf62fe7b401 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:16.127866 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625731b9-5898-44fc-9492-2bf62fe7b401 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:16.129287 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-625731b9-5898-44fc-9492-2bf62fe7b401 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:16.141264 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 611/1218] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Jul 12 17:23:15 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:23:16.147580 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5e3b5a-acff-44e9-8383-e7cc8185c506 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:16.255020 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5e3b5a-acff-44e9-8383-e7cc8185c506 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:16.284217 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:16.290098 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe10ec2-037c-4cac-b606-2b2f86c9bf21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:16.290789 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe10ec2-037c-4cac-b606-2b2f86c9bf21 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:16.291083 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe10ec2-037c-4cac-b606-2b2f86c9bf21 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:16.291366 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe10ec2-037c-4cac-b606-2b2f86c9bf21 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:16.292287 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe10ec2-037c-4cac-b606-2b2f86c9bf21 None admin] RBAC: Adding query filter params (role.id=bef74d4975164f75bcec96ac47afea73, scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2, user.id=f623be28c83247c78d3ea6873c347fe0) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:16.292606 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe10ec2-037c-4cac-b606-2b2f86c9bf21 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:16.294173 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe10ec2-037c-4cac-b606-2b2f86c9bf21 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:16.300495 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 608/1219] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:23:16 2022] GET /identity/v3/role_assignments?user.id=f623be28c83247c78d3ea6873c347fe0&scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2&role.id=bef74d4975164f75bcec96ac47afea73 => generated 259 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:23:18.310413 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaca7ffb-abfc-4185-a50d-11ca5f117057 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:18.310964 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaca7ffb-abfc-4185-a50d-11ca5f117057 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:18.311043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaca7ffb-abfc-4185-a50d-11ca5f117057 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:18.312457 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 612/1220] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:23:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:23:18.317940 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c083b1-370d-40be-8f33-453bf0fd6cf1 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:18.321963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c083b1-370d-40be-8f33-453bf0fd6cf1 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:18.321963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c083b1-370d-40be-8f33-453bf0fd6cf1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:18.335454 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-d8c083b1-370d-40be-8f33-453bf0fd6cf1 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:23:18.336670 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-d8c083b1-370d-40be-8f33-453bf0fd6cf1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:23:18.404596 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 609/1221] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:23:18 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:23:18.428398 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaca7ffb-abfc-4185-a50d-11ca5f117057 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:18.539955 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaca7ffb-abfc-4185-a50d-11ca5f117057 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:18.575323 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2671a143-5b1a-40da-99d8-15ff47d37164 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:18.576047 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2671a143-5b1a-40da-99d8-15ff47d37164 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:18.576335 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2671a143-5b1a-40da-99d8-15ff47d37164 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:18.576617 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2671a143-5b1a-40da-99d8-15ff47d37164 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:18.588462 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2671a143-5b1a-40da-99d8-15ff47d37164 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:23:18.597370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-2671a143-5b1a-40da-99d8-15ff47d37164 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:23:18.597370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:18.597370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:23:18.597370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:23:18.597370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:23:18.597370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:23:18.597370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:23:18.597370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:23:18.597370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:23:18.597370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:23:18.597370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:23:18.597370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:23:18.597370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:23:18.597370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:23:18.597370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:23:18.597370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:23:18.603861 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:23:18.603861 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:23:18.603861 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:23:18.603861 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:23:18.603861 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:23:18.603861 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:23:18.603861 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:23:18.603861 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:23:18.603861 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:23:18.603861 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:23:18.603861 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:23:18.603861 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:23:18.603861 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:23:18.603861 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:23:18.603861 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:23:18.603861 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:23:18.605116 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:23:18.605116 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:23:18.605116 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:23:18.605116 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:23:18.605116 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:23:18.605116 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:23:18.605116 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:23:18.605116 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:23:18.605116 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:23:18.605116 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:23:18.605116 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:23:18.605116 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:23:18.605116 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 613/1222] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Jul 12 17:23:18 2022] GET /identity/v3/roles/service => generated 85 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:23:18.606013 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c083b1-370d-40be-8f33-453bf0fd6cf1 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:18.708269 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c083b1-370d-40be-8f33-453bf0fd6cf1 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:18.732675 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:18.743820 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76743c7f-b38a-4388-a389-3a9283e3bb09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:18.744489 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76743c7f-b38a-4388-a389-3a9283e3bb09 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:18.744773 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76743c7f-b38a-4388-a389-3a9283e3bb09 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:18.745050 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76743c7f-b38a-4388-a389-3a9283e3bb09 None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:18.745916 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76743c7f-b38a-4388-a389-3a9283e3bb09 None admin] RBAC: Adding query filter params (name=service) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:18.746226 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76743c7f-b38a-4388-a389-3a9283e3bb09 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:18.747695 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76743c7f-b38a-4388-a389-3a9283e3bb09 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:18.755652 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 610/1223] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Jul 12 17:23:18 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:23:18.761716 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2671a143-5b1a-40da-99d8-15ff47d37164 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:18.807033 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:18.867121 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2671a143-5b1a-40da-99d8-15ff47d37164 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:18.900831 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb08120c-1fa9-4a59-b6db-c0026c282472 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:18.901623 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb08120c-1fa9-4a59-b6db-c0026c282472 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:18.901912 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb08120c-1fa9-4a59-b6db-c0026c282472 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:18.902194 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb08120c-1fa9-4a59-b6db-c0026c282472 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:18.913726 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb08120c-1fa9-4a59-b6db-c0026c282472 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:18.915344 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb08120c-1fa9-4a59-b6db-c0026c282472 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:18.933379 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-fb08120c-1fa9-4a59-b6db-c0026c282472 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:23:18.933379 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:18.933379 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:23:18.933379 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:23:18.933379 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:23:18.933379 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:23:18.933379 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:23:18.933379 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:23:18.933379 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:23:18.933379 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:23:18.933379 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:23:18.933379 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:23:18.933379 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:23:18.933379 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:23:18.933379 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:23:18.934551 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:23:18.934551 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:23:18.934551 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:23:18.934551 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:23:18.934551 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:23:18.934551 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:23:18.934551 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:23:18.934551 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:23:18.934551 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:23:18.934551 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:23:18.934551 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:23:18.934551 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:23:18.934551 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:23:18.934551 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:23:18.934551 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:23:18.934551 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:23:18.934551 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:23:18.935640 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:23:18.935640 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:23:18.935640 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:23:18.935640 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:23:18.935640 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 614/1224] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:23:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:23:18.941078 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76743c7f-b38a-4388-a389-3a9283e3bb09 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:19.053073 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76743c7f-b38a-4388-a389-3a9283e3bb09 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:19.086606 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2266cb28-e8c6-4b63-922c-3fb6b6ef0195 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:19.087274 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2266cb28-e8c6-4b63-922c-3fb6b6ef0195 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:19.087900 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2266cb28-e8c6-4b63-922c-3fb6b6ef0195 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:19.088159 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2266cb28-e8c6-4b63-922c-3fb6b6ef0195 None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:19.088886 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2266cb28-e8c6-4b63-922c-3fb6b6ef0195 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:19.089201 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2266cb28-e8c6-4b63-922c-3fb6b6ef0195 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:19.090596 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2266cb28-e8c6-4b63-922c-3fb6b6ef0195 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:19.103332 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 611/1225] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:23:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:23:19.109934 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb08120c-1fa9-4a59-b6db-c0026c282472 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:19.204546 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:19.211011 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb08120c-1fa9-4a59-b6db-c0026c282472 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:19.245192 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326f0311-b8c6-4801-be8f-affe785c455f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:19.245990 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326f0311-b8c6-4801-be8f-affe785c455f None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:19.246275 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326f0311-b8c6-4801-be8f-affe785c455f None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:19.246582 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326f0311-b8c6-4801-be8f-affe785c455f None admin] PATH_INFO: `/v3/users/placement` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:19.254354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326f0311-b8c6-4801-be8f-affe785c455f None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:19.255812 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-326f0311-b8c6-4801-be8f-affe785c455f None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:19.269884 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-326f0311-b8c6-4801-be8f-affe785c455f None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 12 17:23:19.269884 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:19.269884 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:23:19.269884 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:23:19.269884 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:23:19.269884 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:23:19.269884 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:23:19.269884 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:23:19.269884 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:23:19.269884 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:23:19.269884 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:23:19.269884 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:23:19.269884 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:23:19.269884 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:23:19.269884 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:23:19.269884 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:23:19.271604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:23:19.271604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:23:19.271604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:23:19.271604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:23:19.271604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:23:19.271604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:23:19.271604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:23:19.271604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:23:19.271604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:23:19.271604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:23:19.271604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:23:19.271604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:23:19.271604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:23:19.271604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:23:19.271604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:23:19.271604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:23:19.271604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:23:19.271604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:23:19.273178 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:23:19.273178 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:23:19.273178 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:23:19.273178 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:23:19.273178 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:23:19.273178 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:23:19.273178 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:23:19.273178 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:23:19.273178 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 12 17:23:19.273178 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:23:19.273178 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 615/1226] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:23:19 2022] GET /identity/v3/users/placement => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:23:19.277297 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2266cb28-e8c6-4b63-922c-3fb6b6ef0195 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:19.326181 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:19.381634 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2266cb28-e8c6-4b63-922c-3fb6b6ef0195 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:19.416915 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129358d2-aede-47e7-a266-3ee45f41a62d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:19.417587 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129358d2-aede-47e7-a266-3ee45f41a62d None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:19.417874 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129358d2-aede-47e7-a266-3ee45f41a62d None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:19.418154 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129358d2-aede-47e7-a266-3ee45f41a62d None admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:19.419021 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129358d2-aede-47e7-a266-3ee45f41a62d None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:19.419334 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129358d2-aede-47e7-a266-3ee45f41a62d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:19.420841 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129358d2-aede-47e7-a266-3ee45f41a62d None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:19.438081 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 612/1227] 10.222.0.16 () {56 vars in 1143 bytes} [Tue Jul 12 17:23:19 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:23:19.443901 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326f0311-b8c6-4801-be8f-affe785c455f None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:19.544213 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326f0311-b8c6-4801-be8f-affe785c455f None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:19.578059 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847fda2e-c843-42a9-a404-25ca734d8870 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:19.578332 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847fda2e-c843-42a9-a404-25ca734d8870 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:19.578752 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847fda2e-c843-42a9-a404-25ca734d8870 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:19.579081 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847fda2e-c843-42a9-a404-25ca734d8870 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:19.589975 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847fda2e-c843-42a9-a404-25ca734d8870 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:19.591843 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847fda2e-c843-42a9-a404-25ca734d8870 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:19.609072 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-847fda2e-c843-42a9-a404-25ca734d8870 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:23:19.609072 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:19.609072 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:23:19.609072 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:23:19.609072 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:23:19.609072 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:23:19.609072 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:23:19.609072 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:23:19.609072 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:23:19.609072 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:23:19.609072 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:23:19.609072 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:23:19.609072 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:23:19.609072 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:23:19.609072 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:23:19.610353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:23:19.610353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:23:19.610353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:23:19.610353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:23:19.610353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:23:19.610353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:23:19.610353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:23:19.610353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:23:19.610353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:23:19.610353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:23:19.610353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:23:19.610353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:23:19.610353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:23:19.610353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:23:19.610353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:23:19.610353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:23:19.610353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:23:19.621818 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:23:19.621818 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:23:19.621818 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:23:19.621818 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:23:19.621818 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 616/1228] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:23:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:23:19.622174 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129358d2-aede-47e7-a266-3ee45f41a62d None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:19.717956 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129358d2-aede-47e7-a266-3ee45f41a62d None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:19.720997 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:19.752341 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9780d14f-0ceb-4667-a3d7-5d1388f833a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:19.753005 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9780d14f-0ceb-4667-a3d7-5d1388f833a4 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:19.753295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9780d14f-0ceb-4667-a3d7-5d1388f833a4 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:19.753579 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9780d14f-0ceb-4667-a3d7-5d1388f833a4 None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:19.754302 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9780d14f-0ceb-4667-a3d7-5d1388f833a4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:19.754648 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9780d14f-0ceb-4667-a3d7-5d1388f833a4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:19.756009 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9780d14f-0ceb-4667-a3d7-5d1388f833a4 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:19.767988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 613/1229] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:23:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:23:19.774254 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847fda2e-c843-42a9-a404-25ca734d8870 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:19.783609 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:19.871348 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847fda2e-c843-42a9-a404-25ca734d8870 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:19.905107 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13525703-fe30-4d46-ab8d-bc151a5eede0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:19.905979 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13525703-fe30-4d46-ab8d-bc151a5eede0 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:19.906271 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13525703-fe30-4d46-ab8d-bc151a5eede0 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:19.906584 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13525703-fe30-4d46-ab8d-bc151a5eede0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:19.914396 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13525703-fe30-4d46-ab8d-bc151a5eede0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:19.917649 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13525703-fe30-4d46-ab8d-bc151a5eede0 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:19.930381 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-13525703-fe30-4d46-ab8d-bc151a5eede0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:23:19.930381 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:19.930381 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:23:19.930381 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:23:19.930381 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:23:19.930381 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:23:19.930381 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:23:19.930381 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:23:19.930381 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:23:19.930381 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:23:19.930381 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:23:19.930381 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:23:19.930381 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:23:19.930381 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:23:19.930381 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:23:19.931503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:23:19.931503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:23:19.931503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:23:19.931503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:23:19.931503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:23:19.931503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:23:19.931503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:23:19.931503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:23:19.931503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:23:19.931503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:23:19.931503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:23:19.931503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:23:19.931503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:23:19.931503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:23:19.931503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:23:19.931503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:23:19.931503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:23:19.932585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:23:19.932585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:23:19.932585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:23:19.932585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:23:19.932585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:23:19.932585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:23:19.932585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:23:19.932585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:23:19.932585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 617/1230] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:23:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:23:19.938040 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9780d14f-0ceb-4667-a3d7-5d1388f833a4 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:20.042525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9780d14f-0ceb-4667-a3d7-5d1388f833a4 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:20.078116 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bcd3514-9208-46d4-850a-e05a56a96a60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:20.078892 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcd3514-9208-46d4-850a-e05a56a96a60 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:20.079185 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcd3514-9208-46d4-850a-e05a56a96a60 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:20.079467 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcd3514-9208-46d4-850a-e05a56a96a60 None admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:20.080263 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bcd3514-9208-46d4-850a-e05a56a96a60 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:20.080577 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bcd3514-9208-46d4-850a-e05a56a96a60 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:20.082014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bcd3514-9208-46d4-850a-e05a56a96a60 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:20.094210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 614/1231] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Jul 12 17:23:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:23:20.100374 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13525703-fe30-4d46-ab8d-bc151a5eede0 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:20.166595 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:20.200855 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13525703-fe30-4d46-ab8d-bc151a5eede0 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:20.236392 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba033f8-9a22-46b1-985f-612fd95d56c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:20.237280 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba033f8-9a22-46b1-985f-612fd95d56c1 None admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:20.237569 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba033f8-9a22-46b1-985f-612fd95d56c1 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:20.237912 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba033f8-9a22-46b1-985f-612fd95d56c1 None admin] PATH_INFO: `/v3/projects/fd4ed2fe037c4fb7ba49c724eb9414c2/users/f623be28c83247c78d3ea6873c347fe0/roles/bef74d4975164f75bcec96ac47afea73` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:20.253119 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ba033f8-9a22-46b1-985f-612fd95d56c1 None admin] RBAC: Authorizing `identity:create_grant(project_id=fd4ed2fe037c4fb7ba49c724eb9414c2, user_id=f623be28c83247c78d3ea6873c347fe0, role_id=bef74d4975164f75bcec96ac47afea73)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:20.254660 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ba033f8-9a22-46b1-985f-612fd95d56c1 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:20.419129 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 618/1232] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Jul 12 17:23:20 2022] PUT /identity/v3/projects/fd4ed2fe037c4fb7ba49c724eb9414c2/users/f623be28c83247c78d3ea6873c347fe0/roles/bef74d4975164f75bcec96ac47afea73 => generated 0 bytes in 319 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:23:22.427427 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb32d33f-3981-457a-83d3-80032816721e None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:22.428046 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb32d33f-3981-457a-83d3-80032816721e None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:22.428046 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb32d33f-3981-457a-83d3-80032816721e None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:22.429482 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 615/1233] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:23:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:23:22.436210 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee0b78a-6962-435f-93fb-3dc1f4bc4b96 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:22.436808 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee0b78a-6962-435f-93fb-3dc1f4bc4b96 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:22.436902 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee0b78a-6962-435f-93fb-3dc1f4bc4b96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:22.454080 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-3ee0b78a-6962-435f-93fb-3dc1f4bc4b96 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:23:22.455494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-3ee0b78a-6962-435f-93fb-3dc1f4bc4b96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:23:22.527894 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 619/1234] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:23:22 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:23:22.552859 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb32d33f-3981-457a-83d3-80032816721e None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:22.580895 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:22.662626 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb32d33f-3981-457a-83d3-80032816721e None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:22.710588 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc160f9e-eeb5-4344-9923-c086bcae339c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:22.711390 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc160f9e-eeb5-4344-9923-c086bcae339c None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:22.711676 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc160f9e-eeb5-4344-9923-c086bcae339c None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:22.711955 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc160f9e-eeb5-4344-9923-c086bcae339c None admin] PATH_INFO: `/v3/roles/service` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:22.724271 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc160f9e-eeb5-4344-9923-c086bcae339c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:23:22.732974 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-cc160f9e-eeb5-4344-9923-c086bcae339c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:23:22.732974 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:22.732974 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:23:22.732974 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:23:22.732974 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:23:22.732974 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:23:22.732974 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:23:22.732974 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:23:22.732974 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:23:22.732974 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:23:22.732974 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:23:22.732974 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:23:22.732974 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:23:22.732974 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:23:22.732974 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:23:22.732974 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:23:22.734843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:23:22.734843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:23:22.734843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:23:22.734843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:23:22.734843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:23:22.734843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:23:22.734843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:23:22.734843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:23:22.734843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:23:22.734843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:23:22.734843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:23:22.734843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:23:22.734843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:23:22.734843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:23:22.734843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:23:22.734843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:23:22.736027 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:23:22.736027 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:23:22.736027 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:23:22.736027 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:23:22.736027 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:23:22.736027 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:23:22.736027 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:23:22.736027 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:23:22.736027 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:23:22.736027 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:23:22.736027 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 12 17:23:22.736027 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:23:22.736027 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 616/1235] 10.222.0.16 () {56 vars in 1102 bytes} [Tue Jul 12 17:23:22 2022] GET /identity/v3/roles/service => generated 85 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:23:22.741362 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ee0b78a-6962-435f-93fb-3dc1f4bc4b96 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:22.789381 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:22.850752 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ee0b78a-6962-435f-93fb-3dc1f4bc4b96 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:22.889371 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efba0d4-5e10-4992-8798-b3b42d7ede0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:22.890790 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efba0d4-5e10-4992-8798-b3b42d7ede0f None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:22.890790 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efba0d4-5e10-4992-8798-b3b42d7ede0f None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:22.890790 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efba0d4-5e10-4992-8798-b3b42d7ede0f None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:22.891436 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4efba0d4-5e10-4992-8798-b3b42d7ede0f None admin] RBAC: Adding query filter params (name=service) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:22.891761 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4efba0d4-5e10-4992-8798-b3b42d7ede0f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:22.893107 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4efba0d4-5e10-4992-8798-b3b42d7ede0f None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:22.902296 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 620/1236] 10.222.0.16 () {56 vars in 1103 bytes} [Tue Jul 12 17:23:22 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:23:22.908814 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc160f9e-eeb5-4344-9923-c086bcae339c None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:22.999222 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:23.017288 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc160f9e-eeb5-4344-9923-c086bcae339c None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:23.052754 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca20e2bb-5d0b-4fb2-9eaa-bd42e0ada764 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:23.053539 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca20e2bb-5d0b-4fb2-9eaa-bd42e0ada764 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:23.053826 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca20e2bb-5d0b-4fb2-9eaa-bd42e0ada764 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:23.054106 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca20e2bb-5d0b-4fb2-9eaa-bd42e0ada764 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:23.063967 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca20e2bb-5d0b-4fb2-9eaa-bd42e0ada764 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:23.065387 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca20e2bb-5d0b-4fb2-9eaa-bd42e0ada764 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:23.083609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-ca20e2bb-5d0b-4fb2-9eaa-bd42e0ada764 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:23:23.083609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:23.083609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:23:23.083609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:23:23.083609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:23:23.083609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:23:23.083609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:23:23.083609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:23:23.083609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:23:23.083609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:23:23.083609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:23:23.083609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:23:23.083609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:23:23.083609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:23:23.083609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:23:23.084719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:23:23.084719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:23:23.084719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:23:23.084719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:23:23.084719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:23:23.084719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:23:23.084719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:23:23.084719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:23:23.084719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:23:23.084719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:23:23.084719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:23:23.084719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:23:23.084719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:23:23.084719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:23:23.084719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:23:23.084719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:23:23.084719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:23:23.085846 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:23:23.085846 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:23:23.085846 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:23:23.085846 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:23:23.085846 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 617/1237] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:23:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:23:23.090868 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efba0d4-5e10-4992-8798-b3b42d7ede0f None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:23.192279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efba0d4-5e10-4992-8798-b3b42d7ede0f None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:23.201593 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:23.226547 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67692158-3b9c-416e-bab7-734c476c0b3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:23.227252 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67692158-3b9c-416e-bab7-734c476c0b3f None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:23.227578 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67692158-3b9c-416e-bab7-734c476c0b3f None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:23.227888 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67692158-3b9c-416e-bab7-734c476c0b3f None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:23.228647 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67692158-3b9c-416e-bab7-734c476c0b3f None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:23.228958 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67692158-3b9c-416e-bab7-734c476c0b3f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:23.230329 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67692158-3b9c-416e-bab7-734c476c0b3f None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:23.243012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 621/1238] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:23:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:23:23.248794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca20e2bb-5d0b-4fb2-9eaa-bd42e0ada764 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:23.351457 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca20e2bb-5d0b-4fb2-9eaa-bd42e0ada764 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:23.386921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7264c8e3-fcc0-42fa-a764-067f8bdd142e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:23.387723 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7264c8e3-fcc0-42fa-a764-067f8bdd142e None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:23.388011 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7264c8e3-fcc0-42fa-a764-067f8bdd142e None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:23.388294 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7264c8e3-fcc0-42fa-a764-067f8bdd142e None admin] PATH_INFO: `/v3/users/placement` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:23.391994 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:23.396816 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7264c8e3-fcc0-42fa-a764-067f8bdd142e None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:23.398250 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7264c8e3-fcc0-42fa-a764-067f8bdd142e None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:23.413762 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-7264c8e3-fcc0-42fa-a764-067f8bdd142e None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 12 17:23:23.413762 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:23.413762 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:23:23.413762 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:23:23.413762 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:23:23.413762 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:23:23.413762 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:23:23.413762 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:23:23.413762 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:23:23.413762 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:23:23.413762 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:23:23.413762 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:23:23.413762 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:23:23.413762 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:23:23.413762 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:23:23.413762 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:23:23.414956 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:23:23.414956 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:23:23.414956 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:23:23.414956 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:23:23.414956 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:23:23.414956 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:23:23.414956 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:23:23.414956 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:23:23.414956 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:23:23.414956 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:23:23.414956 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:23:23.414956 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:23:23.414956 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:23:23.414956 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:23:23.414956 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:23:23.414956 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:23:23.414956 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:23:23.414956 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:23:23.420949 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:23:23.420949 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:23:23.420949 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:23:23.420949 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:23:23.420949 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:23:23.420949 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:23:23.420949 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:23:23.420949 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:23:23.420949 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 12 17:23:23.420949 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:23:23.420949 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 618/1239] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:23:23 2022] GET /identity/v3/users/placement => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:23:23.421834 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67692158-3b9c-416e-bab7-734c476c0b3f None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:23.521918 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67692158-3b9c-416e-bab7-734c476c0b3f None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:23.554879 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cee57c8-95f6-437f-987a-0cac7c8b5f79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:23.555548 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cee57c8-95f6-437f-987a-0cac7c8b5f79 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:23.555829 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cee57c8-95f6-437f-987a-0cac7c8b5f79 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:23.556109 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cee57c8-95f6-437f-987a-0cac7c8b5f79 None admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:23.556940 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cee57c8-95f6-437f-987a-0cac7c8b5f79 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:23.557252 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cee57c8-95f6-437f-987a-0cac7c8b5f79 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:23.558698 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cee57c8-95f6-437f-987a-0cac7c8b5f79 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:23.577675 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 622/1240] 10.222.0.16 () {56 vars in 1143 bytes} [Tue Jul 12 17:23:23 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:23:23.584422 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7264c8e3-fcc0-42fa-a764-067f8bdd142e None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:23.690157 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7264c8e3-fcc0-42fa-a764-067f8bdd142e None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:23.726151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a76461-52a2-420b-809d-7098ed0c624b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:23.726839 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a76461-52a2-420b-809d-7098ed0c624b None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:23.727137 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a76461-52a2-420b-809d-7098ed0c624b None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:23.727473 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a76461-52a2-420b-809d-7098ed0c624b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:23.789172 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:23.798108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a76461-52a2-420b-809d-7098ed0c624b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:23.800170 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a76461-52a2-420b-809d-7098ed0c624b None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:23.899289 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 619/1241] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:23:23 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 12 17:23:23.905291 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cee57c8-95f6-437f-987a-0cac7c8b5f79 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:23.932891 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:24.007938 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cee57c8-95f6-437f-987a-0cac7c8b5f79 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:24.041194 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e0e7f7-bfac-417f-8ec3-c7005516d568 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:24.042070 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e0e7f7-bfac-417f-8ec3-c7005516d568 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:24.042361 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e0e7f7-bfac-417f-8ec3-c7005516d568 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:24.042672 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e0e7f7-bfac-417f-8ec3-c7005516d568 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:24.053164 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7e0e7f7-bfac-417f-8ec3-c7005516d568 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:24.054640 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7e0e7f7-bfac-417f-8ec3-c7005516d568 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:24.072189 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-f7e0e7f7-bfac-417f-8ec3-c7005516d568 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:23:24.072189 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:24.072189 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:23:24.072189 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:23:24.072189 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:23:24.072189 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:23:24.072189 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:23:24.072189 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:23:24.072189 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:23:24.072189 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:23:24.072189 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:23:24.072189 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:23:24.072189 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:23:24.072189 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:23:24.072189 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:23:24.075354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:23:24.075354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:23:24.075354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:23:24.075354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:23:24.075354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:23:24.075354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:23:24.075354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:23:24.075354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:23:24.075354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:23:24.075354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:23:24.075354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:23:24.075354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:23:24.075354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:23:24.075354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:23:24.075354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:23:24.075354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:23:24.075354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:23:24.076267 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:23:24.076267 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:23:24.076267 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:23:24.076267 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:23:24.076267 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 623/1242] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:23:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:23:24.079363 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a76461-52a2-420b-809d-7098ed0c624b None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:24.174133 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:24.179601 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a76461-52a2-420b-809d-7098ed0c624b None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:24.214350 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a52758c-7d3e-4e60-b235-57cc743d5e9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:24.215075 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a52758c-7d3e-4e60-b235-57cc743d5e9a None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:24.215671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a52758c-7d3e-4e60-b235-57cc743d5e9a None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:24.215960 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a52758c-7d3e-4e60-b235-57cc743d5e9a None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:24.216684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a52758c-7d3e-4e60-b235-57cc743d5e9a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:24.216997 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a52758c-7d3e-4e60-b235-57cc743d5e9a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:24.219137 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a52758c-7d3e-4e60-b235-57cc743d5e9a None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:24.230552 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 620/1243] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:23:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:23:24.237030 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e0e7f7-bfac-417f-8ec3-c7005516d568 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:24.316325 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:24.338336 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e0e7f7-bfac-417f-8ec3-c7005516d568 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:24.372615 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0dcda33-8793-4757-bde3-00a6629f2b8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:24.373462 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0dcda33-8793-4757-bde3-00a6629f2b8e None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:24.373754 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0dcda33-8793-4757-bde3-00a6629f2b8e None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:24.374060 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0dcda33-8793-4757-bde3-00a6629f2b8e None admin] PATH_INFO: `/v3/projects/service` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:24.380784 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0dcda33-8793-4757-bde3-00a6629f2b8e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:24.382342 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0dcda33-8793-4757-bde3-00a6629f2b8e None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:24.396976 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-b0dcda33-8793-4757-bde3-00a6629f2b8e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:23:24.396976 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:24.396976 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:23:24.396976 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:23:24.396976 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:23:24.396976 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:23:24.396976 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:23:24.396976 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:23:24.396976 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:23:24.396976 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:23:24.396976 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:23:24.396976 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:23:24.396976 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:23:24.396976 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:23:24.396976 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:23:24.405509 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:23:24.405509 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:23:24.405509 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:23:24.405509 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:23:24.405509 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:23:24.405509 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:23:24.405509 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:23:24.405509 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:23:24.405509 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:23:24.405509 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:23:24.405509 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:23:24.405509 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:23:24.405509 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:23:24.405509 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:23:24.405509 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:23:24.405509 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:23:24.405509 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:23:24.406819 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:23:24.406819 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:23:24.406819 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:23:24.406819 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:23:24.406819 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:23:24.406819 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:23:24.406819 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:23:24.406819 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:23:24.406819 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 624/1244] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:23:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:23:24.407404 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a52758c-7d3e-4e60-b235-57cc743d5e9a None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:24.507274 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a52758c-7d3e-4e60-b235-57cc743d5e9a None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:24.541516 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d432d35a-046e-421f-8ed1-604c5d393ea0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:24.542181 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d432d35a-046e-421f-8ed1-604c5d393ea0 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:24.542494 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d432d35a-046e-421f-8ed1-604c5d393ea0 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:24.542783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d432d35a-046e-421f-8ed1-604c5d393ea0 None admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:24.543620 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d432d35a-046e-421f-8ed1-604c5d393ea0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:24.543933 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d432d35a-046e-421f-8ed1-604c5d393ea0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:24.545458 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d432d35a-046e-421f-8ed1-604c5d393ea0 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:24.558099 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 621/1245] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Jul 12 17:23:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:23:24.564365 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0dcda33-8793-4757-bde3-00a6629f2b8e None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:24.675883 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0dcda33-8793-4757-bde3-00a6629f2b8e None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:24.711216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b46eec5-5d9b-4b5a-a1bc-3d6c3194f863 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:24.711874 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b46eec5-5d9b-4b5a-a1bc-3d6c3194f863 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:24.712413 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b46eec5-5d9b-4b5a-a1bc-3d6c3194f863 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:24.712736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b46eec5-5d9b-4b5a-a1bc-3d6c3194f863 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:24.713572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b46eec5-5d9b-4b5a-a1bc-3d6c3194f863 None admin] RBAC: Adding query filter params (role.id=bef74d4975164f75bcec96ac47afea73, scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2, user.id=f623be28c83247c78d3ea6873c347fe0) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:24.713885 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b46eec5-5d9b-4b5a-a1bc-3d6c3194f863 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:24.715433 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b46eec5-5d9b-4b5a-a1bc-3d6c3194f863 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:24.722362 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 625/1246] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:23:24 2022] GET /identity/v3/role_assignments?user.id=f623be28c83247c78d3ea6873c347fe0&scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2&role.id=bef74d4975164f75bcec96ac47afea73 => generated 608 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:23:26.785447 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec712fe-42d2-4037-870d-76f3866d2be3 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:26.785999 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec712fe-42d2-4037-870d-76f3866d2be3 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:26.785999 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec712fe-42d2-4037-870d-76f3866d2be3 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:26.787382 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 622/1247] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:23:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:23:26.792445 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f778a5-513a-4136-91a0-1dbe65b7cb80 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:26.792931 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f778a5-513a-4136-91a0-1dbe65b7cb80 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:26.793120 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f778a5-513a-4136-91a0-1dbe65b7cb80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:26.799085 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:26.811823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-24f778a5-513a-4136-91a0-1dbe65b7cb80 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:23:26.813032 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-24f778a5-513a-4136-91a0-1dbe65b7cb80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:23:26.881011 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 626/1248] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:23:26 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:23:26.905839 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec712fe-42d2-4037-870d-76f3866d2be3 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:26.924311 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:27.009621 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec712fe-42d2-4037-870d-76f3866d2be3 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:27.045700 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a5bac1-ba06-4728-af6b-ca6648d81163 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:27.046593 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a5bac1-ba06-4728-af6b-ca6648d81163 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:27.046885 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a5bac1-ba06-4728-af6b-ca6648d81163 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:27.047183 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a5bac1-ba06-4728-af6b-ca6648d81163 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:27.057314 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a5bac1-ba06-4728-af6b-ca6648d81163 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:23:27.065110 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-46a5bac1-ba06-4728-af6b-ca6648d81163 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:23:27.065110 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:27.065110 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:23:27.065110 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:23:27.065110 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:23:27.065110 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:23:27.065110 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:23:27.065110 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:23:27.065110 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:23:27.065110 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:23:27.065110 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:23:27.065110 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:23:27.065110 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:23:27.065110 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:23:27.065110 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:23:27.065110 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:23:27.066124 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:23:27.066124 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:23:27.066124 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:23:27.066124 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:23:27.066124 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:23:27.066124 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:23:27.066124 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:23:27.066124 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:23:27.066124 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:23:27.066124 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:23:27.066124 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:23:27.066124 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:23:27.066124 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:23:27.066124 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:23:27.066124 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:23:27.066124 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:23:27.067170 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:23:27.067170 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:23:27.067170 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:23:27.067170 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:23:27.067170 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:23:27.067170 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:23:27.067170 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:23:27.067170 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:23:27.067170 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:23:27.067170 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:23:27.067170 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:23:27.067170 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:23:27.067170 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 623/1249] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Jul 12 17:23:26 2022] GET /identity/v3/roles/admin => generated 83 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:23:27.072446 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f778a5-513a-4136-91a0-1dbe65b7cb80 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:27.172459 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f778a5-513a-4136-91a0-1dbe65b7cb80 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:27.200912 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:27.206220 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a719c646-54f1-4086-a412-0d725beec709 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:27.206948 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a719c646-54f1-4086-a412-0d725beec709 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:27.207277 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a719c646-54f1-4086-a412-0d725beec709 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:27.207568 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a719c646-54f1-4086-a412-0d725beec709 None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:27.208326 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a719c646-54f1-4086-a412-0d725beec709 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:27.208642 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a719c646-54f1-4086-a412-0d725beec709 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:27.210020 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a719c646-54f1-4086-a412-0d725beec709 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:27.219683 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 627/1250] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:23:27 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:23:27.225560 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a5bac1-ba06-4728-af6b-ca6648d81163 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:27.304166 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:27.326864 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a5bac1-ba06-4728-af6b-ca6648d81163 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:27.360628 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df15e15-dbfa-4fee-93f8-4c95467859ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:27.361479 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df15e15-dbfa-4fee-93f8-4c95467859ae None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:27.361765 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df15e15-dbfa-4fee-93f8-4c95467859ae None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:27.362111 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df15e15-dbfa-4fee-93f8-4c95467859ae None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:27.373024 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df15e15-dbfa-4fee-93f8-4c95467859ae None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:27.374580 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df15e15-dbfa-4fee-93f8-4c95467859ae None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:27.391449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-5df15e15-dbfa-4fee-93f8-4c95467859ae None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:23:27.391449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:27.391449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:23:27.391449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:23:27.391449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:23:27.391449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:23:27.391449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:23:27.391449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:23:27.391449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:23:27.391449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:23:27.391449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:23:27.391449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:23:27.391449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:23:27.391449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:23:27.391449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:23:27.392546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:23:27.392546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:23:27.392546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:23:27.392546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:23:27.392546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:23:27.392546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:23:27.392546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:23:27.392546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:23:27.392546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:23:27.392546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:23:27.392546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:23:27.392546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:23:27.392546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:23:27.392546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:23:27.392546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:23:27.392546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:23:27.392546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:23:27.393682 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:23:27.393682 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:23:27.393682 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:23:27.393682 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:23:27.393682 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 624/1251] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:23:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:23:27.398480 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a719c646-54f1-4086-a412-0d725beec709 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:27.500586 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a719c646-54f1-4086-a412-0d725beec709 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:27.533463 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b82bfb-e34f-40ef-b23b-46c39f4e3bf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:27.534121 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b82bfb-e34f-40ef-b23b-46c39f4e3bf7 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:27.534406 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b82bfb-e34f-40ef-b23b-46c39f4e3bf7 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:27.534711 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b82bfb-e34f-40ef-b23b-46c39f4e3bf7 None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:27.535428 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b82bfb-e34f-40ef-b23b-46c39f4e3bf7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:27.535738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b82bfb-e34f-40ef-b23b-46c39f4e3bf7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:27.537095 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b82bfb-e34f-40ef-b23b-46c39f4e3bf7 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:27.549389 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 628/1252] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:23:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:23:27.555038 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df15e15-dbfa-4fee-93f8-4c95467859ae None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:27.659944 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df15e15-dbfa-4fee-93f8-4c95467859ae None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:27.686023 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:27.693870 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125d5447-40de-4ff8-a965-ca0a04651b4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:27.694718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125d5447-40de-4ff8-a965-ca0a04651b4e None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:27.695011 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125d5447-40de-4ff8-a965-ca0a04651b4e None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:27.695340 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125d5447-40de-4ff8-a965-ca0a04651b4e None admin] PATH_INFO: `/v3/users/placement` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:27.703546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-125d5447-40de-4ff8-a965-ca0a04651b4e None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:27.705002 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-125d5447-40de-4ff8-a965-ca0a04651b4e None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:27.720078 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-125d5447-40de-4ff8-a965-ca0a04651b4e None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 12 17:23:27.720078 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:27.720078 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:23:27.720078 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:23:27.720078 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:23:27.720078 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:23:27.720078 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:23:27.720078 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:23:27.720078 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:23:27.720078 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:23:27.720078 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:23:27.720078 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:23:27.720078 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:23:27.720078 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:23:27.720078 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:23:27.720078 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:23:27.721320 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:23:27.721320 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:23:27.721320 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:23:27.721320 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:23:27.721320 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:23:27.721320 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:23:27.721320 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:23:27.721320 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:23:27.721320 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:23:27.721320 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:23:27.721320 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:23:27.721320 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:23:27.721320 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:23:27.721320 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:23:27.721320 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:23:27.721320 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:23:27.721320 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:23:27.721320 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:23:27.722291 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:23:27.722291 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:23:27.722291 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:23:27.722291 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:23:27.722291 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:23:27.722291 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:23:27.722291 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:23:27.722291 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:23:27.722291 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 12 17:23:27.722291 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:23:27.722291 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 625/1253] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:23:27 2022] GET /identity/v3/users/placement => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:23:27.727412 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b82bfb-e34f-40ef-b23b-46c39f4e3bf7 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:27.781965 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:27.826269 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b82bfb-e34f-40ef-b23b-46c39f4e3bf7 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:27.862668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b77669-61ca-4387-9573-9ab460dc1a7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:27.863339 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b77669-61ca-4387-9573-9ab460dc1a7e None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:27.863628 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b77669-61ca-4387-9573-9ab460dc1a7e None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:27.863915 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b77669-61ca-4387-9573-9ab460dc1a7e None admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:27.864781 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b77669-61ca-4387-9573-9ab460dc1a7e None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:27.865096 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b77669-61ca-4387-9573-9ab460dc1a7e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:27.866644 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b77669-61ca-4387-9573-9ab460dc1a7e None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:27.884681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 629/1254] 10.222.0.16 () {56 vars in 1143 bytes} [Tue Jul 12 17:23:27 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:23:27.890657 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125d5447-40de-4ff8-a965-ca0a04651b4e None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:28.001074 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125d5447-40de-4ff8-a965-ca0a04651b4e None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:28.039640 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-457b4d92-096a-4ad9-8c96-32dc1557e8e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:28.040236 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457b4d92-096a-4ad9-8c96-32dc1557e8e6 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:28.040530 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457b4d92-096a-4ad9-8c96-32dc1557e8e6 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:28.040820 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457b4d92-096a-4ad9-8c96-32dc1557e8e6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:28.106331 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:28.120542 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-457b4d92-096a-4ad9-8c96-32dc1557e8e6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:28.122193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-457b4d92-096a-4ad9-8c96-32dc1557e8e6 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:28.231537 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 626/1255] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:23:27 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 12 17:23:28.238456 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b77669-61ca-4387-9573-9ab460dc1a7e None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:28.348186 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b77669-61ca-4387-9573-9ab460dc1a7e None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:28.362852 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:28.384500 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e9b582-5d14-4dfe-bead-ace5ae23f542 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:28.385295 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e9b582-5d14-4dfe-bead-ace5ae23f542 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:28.385585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e9b582-5d14-4dfe-bead-ace5ae23f542 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:28.385871 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e9b582-5d14-4dfe-bead-ace5ae23f542 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:28.397194 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e9b582-5d14-4dfe-bead-ace5ae23f542 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:28.398663 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e9b582-5d14-4dfe-bead-ace5ae23f542 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:28.417410 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-f2e9b582-5d14-4dfe-bead-ace5ae23f542 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:23:28.417410 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:28.417410 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:23:28.417410 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:23:28.417410 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:23:28.417410 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:23:28.417410 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:23:28.417410 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:23:28.417410 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:23:28.417410 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:23:28.417410 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:23:28.417410 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:23:28.417410 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:23:28.417410 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:23:28.417410 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:23:28.419297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:23:28.419297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:23:28.419297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:23:28.419297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:23:28.419297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:23:28.419297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:23:28.419297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:23:28.419297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:23:28.419297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:23:28.419297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:23:28.419297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:23:28.419297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:23:28.419297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:23:28.419297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:23:28.419297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:23:28.419297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:23:28.419297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:23:28.420395 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:23:28.420395 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:23:28.420395 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:23:28.420395 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:23:28.420395 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 630/1256] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:23:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:23:28.424861 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-457b4d92-096a-4ad9-8c96-32dc1557e8e6 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:28.515297 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:28.527724 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-457b4d92-096a-4ad9-8c96-32dc1557e8e6 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:28.564586 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc3357b-1e2f-4182-8ced-70e70cb57c13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:28.565301 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc3357b-1e2f-4182-8ced-70e70cb57c13 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:28.565592 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc3357b-1e2f-4182-8ced-70e70cb57c13 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:28.565878 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bc3357b-1e2f-4182-8ced-70e70cb57c13 None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:28.566765 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc3357b-1e2f-4182-8ced-70e70cb57c13 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:28.567023 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc3357b-1e2f-4182-8ced-70e70cb57c13 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:28.568405 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bc3357b-1e2f-4182-8ced-70e70cb57c13 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:28.581314 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 627/1257] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:23:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:23:28.587163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e9b582-5d14-4dfe-bead-ace5ae23f542 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:28.692239 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e9b582-5d14-4dfe-bead-ace5ae23f542 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:28.727508 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d22302-993e-4674-81e9-a33dc7ef3918 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:28.728308 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d22302-993e-4674-81e9-a33dc7ef3918 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:28.728589 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d22302-993e-4674-81e9-a33dc7ef3918 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:28.728939 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d22302-993e-4674-81e9-a33dc7ef3918 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:28.736611 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d22302-993e-4674-81e9-a33dc7ef3918 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:28.738123 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d22302-993e-4674-81e9-a33dc7ef3918 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:28.753392 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-17d22302-993e-4674-81e9-a33dc7ef3918 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:23:28.753392 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:28.753392 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:23:28.753392 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:23:28.753392 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:23:28.753392 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:23:28.753392 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:23:28.753392 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:23:28.753392 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:23:28.753392 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:23:28.753392 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:23:28.753392 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:23:28.753392 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:23:28.753392 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:23:28.753392 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:23:28.754570 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:23:28.754570 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:23:28.754570 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:23:28.754570 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:23:28.754570 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:23:28.754570 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:23:28.754570 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:23:28.754570 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:23:28.754570 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:23:28.754570 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:23:28.754570 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:23:28.754570 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:23:28.754570 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:23:28.754570 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:23:28.754570 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:23:28.754570 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:23:28.754570 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:23:28.755803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:23:28.755803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:23:28.755803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:23:28.755803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:23:28.755803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:23:28.755803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:23:28.755803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:23:28.755803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:23:28.755803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 631/1258] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:23:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:23:28.761060 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc3357b-1e2f-4182-8ced-70e70cb57c13 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:28.864837 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bc3357b-1e2f-4182-8ced-70e70cb57c13 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:28.899583 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c84913aa-8de3-4366-8521-09a090cddc1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:28.900257 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84913aa-8de3-4366-8521-09a090cddc1b None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:28.900525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84913aa-8de3-4366-8521-09a090cddc1b None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:28.900905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c84913aa-8de3-4366-8521-09a090cddc1b None admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:28.901693 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c84913aa-8de3-4366-8521-09a090cddc1b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:28.902011 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c84913aa-8de3-4366-8521-09a090cddc1b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:28.903564 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c84913aa-8de3-4366-8521-09a090cddc1b None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:28.915891 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 628/1259] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Jul 12 17:23:28 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:23:28.922125 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d22302-993e-4674-81e9-a33dc7ef3918 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:28.948955 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:29.027351 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d22302-993e-4674-81e9-a33dc7ef3918 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:29.062051 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e64de1-9aef-496d-a323-92a5436b394b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:29.062772 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e64de1-9aef-496d-a323-92a5436b394b None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:29.063595 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e64de1-9aef-496d-a323-92a5436b394b None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:29.063877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e64de1-9aef-496d-a323-92a5436b394b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:29.064837 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e64de1-9aef-496d-a323-92a5436b394b None admin] RBAC: Adding query filter params (role.id=eb05bf286c894ea9bf44a6c023a4323b, scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2, user.id=f623be28c83247c78d3ea6873c347fe0) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:29.065155 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e64de1-9aef-496d-a323-92a5436b394b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:29.066656 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e64de1-9aef-496d-a323-92a5436b394b None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:29.072196 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 632/1260] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:23:28 2022] GET /identity/v3/role_assignments?user.id=f623be28c83247c78d3ea6873c347fe0&scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2&role.id=eb05bf286c894ea9bf44a6c023a4323b => generated 259 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:23:31.115365 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38030aa0-ea55-4605-b355-c7b69f0625d1 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:31.116054 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38030aa0-ea55-4605-b355-c7b69f0625d1 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:31.116054 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38030aa0-ea55-4605-b355-c7b69f0625d1 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:31.117334 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 629/1261] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:23:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:23:31.123435 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f959f661-552f-46f4-9db5-93e17c71cfb9 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:31.123909 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f959f661-552f-46f4-9db5-93e17c71cfb9 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:31.124003 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f959f661-552f-46f4-9db5-93e17c71cfb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:31.142253 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-f959f661-552f-46f4-9db5-93e17c71cfb9 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:23:31.143504 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-f959f661-552f-46f4-9db5-93e17c71cfb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:23:31.211090 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 633/1262] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:23:31 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:23:31.234229 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38030aa0-ea55-4605-b355-c7b69f0625d1 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:31.245415 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:31.338055 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38030aa0-ea55-4605-b355-c7b69f0625d1 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:31.372316 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8543b349-db25-4c85-a7a8-be14350f99ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:31.373080 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8543b349-db25-4c85-a7a8-be14350f99ac None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:31.373576 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8543b349-db25-4c85-a7a8-be14350f99ac None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:31.374018 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8543b349-db25-4c85-a7a8-be14350f99ac None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:31.385045 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8543b349-db25-4c85-a7a8-be14350f99ac None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:23:31.393681 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-8543b349-db25-4c85-a7a8-be14350f99ac None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:23:31.393681 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:31.393681 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:23:31.393681 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:23:31.393681 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:23:31.393681 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:23:31.393681 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:23:31.393681 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:23:31.393681 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:23:31.393681 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:23:31.393681 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:23:31.393681 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:23:31.393681 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:23:31.393681 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:23:31.393681 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:23:31.393681 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:23:31.394854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:23:31.394854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:23:31.394854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:23:31.394854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:23:31.394854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:23:31.394854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:23:31.394854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:23:31.394854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:23:31.394854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:23:31.394854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:23:31.394854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:23:31.394854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:23:31.394854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:23:31.394854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:23:31.394854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:23:31.394854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:23:31.395957 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:23:31.395957 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:23:31.395957 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:23:31.395957 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:23:31.395957 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:23:31.395957 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:23:31.395957 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:23:31.395957 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:23:31.395957 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:23:31.395957 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:23:31.395957 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:23:31.395957 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:23:31.395957 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 630/1263] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Jul 12 17:23:31 2022] GET /identity/v3/roles/admin => generated 83 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:23:31.401612 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f959f661-552f-46f4-9db5-93e17c71cfb9 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:31.421378 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:31.510822 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f959f661-552f-46f4-9db5-93e17c71cfb9 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:31.549873 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6390f1d1-85b1-4906-9a6b-6405a7983a21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:31.550583 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6390f1d1-85b1-4906-9a6b-6405a7983a21 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:31.550876 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6390f1d1-85b1-4906-9a6b-6405a7983a21 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:31.551161 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6390f1d1-85b1-4906-9a6b-6405a7983a21 None admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:31.551921 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6390f1d1-85b1-4906-9a6b-6405a7983a21 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:31.552265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6390f1d1-85b1-4906-9a6b-6405a7983a21 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:31.553768 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6390f1d1-85b1-4906-9a6b-6405a7983a21 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:31.562343 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 634/1264] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:23:31 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:23:31.568764 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8543b349-db25-4c85-a7a8-be14350f99ac None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:31.637480 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:31.670929 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8543b349-db25-4c85-a7a8-be14350f99ac None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:31.705504 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206d740d-9462-40c7-84c4-dd4d898bbbb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:31.706297 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206d740d-9462-40c7-84c4-dd4d898bbbb4 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:31.706630 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206d740d-9462-40c7-84c4-dd4d898bbbb4 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:31.706914 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206d740d-9462-40c7-84c4-dd4d898bbbb4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:31.719822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206d740d-9462-40c7-84c4-dd4d898bbbb4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:31.721861 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206d740d-9462-40c7-84c4-dd4d898bbbb4 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:31.740502 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-206d740d-9462-40c7-84c4-dd4d898bbbb4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:23:31.740502 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:31.740502 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:23:31.740502 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:23:31.740502 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:23:31.740502 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:23:31.740502 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:23:31.740502 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:23:31.740502 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:23:31.740502 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:23:31.740502 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:23:31.740502 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:23:31.740502 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:23:31.740502 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:23:31.740502 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:23:31.741623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:23:31.741623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:23:31.741623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:23:31.741623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:23:31.741623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:23:31.741623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:23:31.741623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:23:31.741623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:23:31.741623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:23:31.741623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:23:31.741623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:23:31.741623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:23:31.741623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:23:31.741623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:23:31.741623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:23:31.741623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:23:31.741623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:23:31.742877 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:23:31.742877 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:23:31.742877 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:23:31.742877 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:23:31.742877 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 631/1265] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:23:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:23:31.747639 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6390f1d1-85b1-4906-9a6b-6405a7983a21 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:31.831933 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:31.850709 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6390f1d1-85b1-4906-9a6b-6405a7983a21 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:31.883782 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53dd1b5b-eb10-4170-b08b-e5354174ce32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:31.884439 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53dd1b5b-eb10-4170-b08b-e5354174ce32 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:31.884727 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53dd1b5b-eb10-4170-b08b-e5354174ce32 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:31.885009 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53dd1b5b-eb10-4170-b08b-e5354174ce32 None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:31.885726 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53dd1b5b-eb10-4170-b08b-e5354174ce32 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:31.886035 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53dd1b5b-eb10-4170-b08b-e5354174ce32 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:31.887430 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53dd1b5b-eb10-4170-b08b-e5354174ce32 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:31.899389 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 635/1266] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:23:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:23:31.905314 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206d740d-9462-40c7-84c4-dd4d898bbbb4 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:32.017573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206d740d-9462-40c7-84c4-dd4d898bbbb4 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:32.040328 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:32.056456 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de36951d-d976-4b3c-9831-b3f3b86b71d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:32.057356 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de36951d-d976-4b3c-9831-b3f3b86b71d9 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:32.057691 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de36951d-d976-4b3c-9831-b3f3b86b71d9 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:32.058015 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de36951d-d976-4b3c-9831-b3f3b86b71d9 None admin] PATH_INFO: `/v3/users/placement` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:32.067550 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de36951d-d976-4b3c-9831-b3f3b86b71d9 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:32.069046 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de36951d-d976-4b3c-9831-b3f3b86b71d9 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:32.086310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-de36951d-d976-4b3c-9831-b3f3b86b71d9 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 12 17:23:32.086310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:32.086310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:23:32.086310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:23:32.086310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:23:32.086310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:23:32.086310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:23:32.086310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:23:32.086310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:23:32.086310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:23:32.086310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:23:32.086310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:23:32.086310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:23:32.086310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:23:32.086310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:23:32.086310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:23:32.087792 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:23:32.087792 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:23:32.087792 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:23:32.087792 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:23:32.087792 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:23:32.087792 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:23:32.087792 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:23:32.087792 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:23:32.087792 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:23:32.087792 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:23:32.087792 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:23:32.087792 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:23:32.087792 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:23:32.087792 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:23:32.087792 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:23:32.087792 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:23:32.087792 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:23:32.087792 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:23:32.088843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:23:32.088843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:23:32.088843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:23:32.088843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:23:32.088843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:23:32.088843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:23:32.088843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:23:32.088843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:23:32.088843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 12 17:23:32.088843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:23:32.089427 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 632/1267] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:23:31 2022] GET /identity/v3/users/placement => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:23:32.095129 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53dd1b5b-eb10-4170-b08b-e5354174ce32 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:32.207169 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53dd1b5b-eb10-4170-b08b-e5354174ce32 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:32.246720 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c1dfcb-3836-43cd-ba51-e0e66c707f3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:32.247406 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c1dfcb-3836-43cd-ba51-e0e66c707f3f None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:32.247690 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c1dfcb-3836-43cd-ba51-e0e66c707f3f None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:32.247967 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c1dfcb-3836-43cd-ba51-e0e66c707f3f None admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:32.248825 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90c1dfcb-3836-43cd-ba51-e0e66c707f3f None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:32.249142 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90c1dfcb-3836-43cd-ba51-e0e66c707f3f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:32.251434 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90c1dfcb-3836-43cd-ba51-e0e66c707f3f None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:32.261377 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:32.271047 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 636/1268] 10.222.0.16 () {56 vars in 1143 bytes} [Tue Jul 12 17:23:32 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:23:32.277026 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de36951d-d976-4b3c-9831-b3f3b86b71d9 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:32.391807 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de36951d-d976-4b3c-9831-b3f3b86b71d9 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:32.428617 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3389f5-c02e-43da-ab9d-61c6d3a6caa5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:32.429456 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3389f5-c02e-43da-ab9d-61c6d3a6caa5 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:32.429750 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3389f5-c02e-43da-ab9d-61c6d3a6caa5 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:32.430035 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3389f5-c02e-43da-ab9d-61c6d3a6caa5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:32.441660 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c3389f5-c02e-43da-ab9d-61c6d3a6caa5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:32.443163 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c3389f5-c02e-43da-ab9d-61c6d3a6caa5 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:32.461896 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-9c3389f5-c02e-43da-ab9d-61c6d3a6caa5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:23:32.461896 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:32.461896 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:23:32.461896 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:23:32.461896 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:23:32.461896 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:23:32.461896 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:23:32.461896 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:23:32.461896 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:23:32.461896 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:23:32.461896 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:23:32.461896 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:23:32.461896 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:23:32.461896 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:23:32.461896 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:23:32.463055 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:23:32.463055 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:23:32.463055 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:23:32.463055 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:23:32.463055 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:23:32.463055 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:23:32.463055 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:23:32.463055 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:23:32.463055 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:23:32.463055 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:23:32.463055 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:23:32.463055 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:23:32.463055 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:23:32.463055 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:23:32.463055 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:23:32.463055 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:23:32.463055 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:23:32.464186 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:23:32.464186 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:23:32.464186 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:23:32.464186 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:23:32.464186 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 633/1269] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:23:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:23:32.468979 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c1dfcb-3836-43cd-ba51-e0e66c707f3f None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:32.570699 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c1dfcb-3836-43cd-ba51-e0e66c707f3f None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:32.606617 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee47a72-cd21-43ac-899e-181927d23989 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:32.607354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee47a72-cd21-43ac-899e-181927d23989 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:32.607651 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee47a72-cd21-43ac-899e-181927d23989 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:32.607936 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee47a72-cd21-43ac-899e-181927d23989 None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:32.608664 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee47a72-cd21-43ac-899e-181927d23989 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:32.608984 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee47a72-cd21-43ac-899e-181927d23989 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:32.610565 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee47a72-cd21-43ac-899e-181927d23989 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:32.622046 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 637/1270] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:23:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:23:32.628472 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3389f5-c02e-43da-ab9d-61c6d3a6caa5 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:32.661114 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:32.729848 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3389f5-c02e-43da-ab9d-61c6d3a6caa5 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:32.764061 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0942541f-31ec-40b9-83bf-85dc4e089a8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:32.764923 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0942541f-31ec-40b9-83bf-85dc4e089a8d None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:32.765213 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0942541f-31ec-40b9-83bf-85dc4e089a8d None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:32.765505 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0942541f-31ec-40b9-83bf-85dc4e089a8d None admin] PATH_INFO: `/v3/projects/service` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:32.772897 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0942541f-31ec-40b9-83bf-85dc4e089a8d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:32.774343 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0942541f-31ec-40b9-83bf-85dc4e089a8d None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:32.788012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-0942541f-31ec-40b9-83bf-85dc4e089a8d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:23:32.788012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:32.788012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:23:32.788012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:23:32.788012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:23:32.788012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:23:32.788012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:23:32.788012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:23:32.788012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:23:32.788012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:23:32.788012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:23:32.788012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:23:32.788012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:23:32.788012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:23:32.788012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:23:32.789047 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:23:32.789047 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:23:32.789047 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:23:32.789047 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:23:32.789047 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:23:32.789047 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:23:32.789047 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:23:32.789047 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:23:32.789047 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:23:32.789047 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:23:32.789047 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:23:32.789047 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:23:32.789047 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:23:32.789047 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:23:32.789047 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:23:32.789047 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:23:32.789047 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:23:32.790359 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:23:32.790359 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:23:32.790359 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:23:32.790359 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:23:32.790359 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:23:32.790359 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:23:32.790359 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:23:32.790359 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:23:32.790359 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 634/1271] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:23:32 2022] GET /identity/v3/projects/service => generated 88 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:23:32.795166 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee47a72-cd21-43ac-899e-181927d23989 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:32.854418 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:32.895229 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee47a72-cd21-43ac-899e-181927d23989 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:32.928792 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce074ca0-370c-47d3-b92c-5f3531763fc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:32.929444 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce074ca0-370c-47d3-b92c-5f3531763fc4 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:32.929729 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce074ca0-370c-47d3-b92c-5f3531763fc4 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:32.930008 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce074ca0-370c-47d3-b92c-5f3531763fc4 None admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:32.930865 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce074ca0-370c-47d3-b92c-5f3531763fc4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:32.931177 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce074ca0-370c-47d3-b92c-5f3531763fc4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:32.932595 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce074ca0-370c-47d3-b92c-5f3531763fc4 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:32.944890 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 638/1272] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Jul 12 17:23:32 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:23:32.950820 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0942541f-31ec-40b9-83bf-85dc4e089a8d None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:33.044010 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:33.053684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0942541f-31ec-40b9-83bf-85dc4e089a8d None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:33.089313 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b4eed22-5561-4838-a121-3ac9ff2a7597 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:33.090277 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4eed22-5561-4838-a121-3ac9ff2a7597 None admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:33.096484 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4eed22-5561-4838-a121-3ac9ff2a7597 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:33.096484 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4eed22-5561-4838-a121-3ac9ff2a7597 None admin] PATH_INFO: `/v3/projects/fd4ed2fe037c4fb7ba49c724eb9414c2/users/f623be28c83247c78d3ea6873c347fe0/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:33.096484 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b4eed22-5561-4838-a121-3ac9ff2a7597 None admin] RBAC: Authorizing `identity:create_grant(project_id=fd4ed2fe037c4fb7ba49c724eb9414c2, user_id=f623be28c83247c78d3ea6873c347fe0, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:33.096484 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b4eed22-5561-4838-a121-3ac9ff2a7597 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:33.206297 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 635/1273] 10.222.0.16 () {58 vars in 1439 bytes} [Tue Jul 12 17:23:32 2022] PUT /identity/v3/projects/fd4ed2fe037c4fb7ba49c724eb9414c2/users/f623be28c83247c78d3ea6873c347fe0/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:23:35.254906 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a73cd0-ac55-445e-aef3-391cf37c93cf None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:35.255437 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a73cd0-ac55-445e-aef3-391cf37c93cf None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:35.255553 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a73cd0-ac55-445e-aef3-391cf37c93cf None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:35.256906 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 639/1274] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:23:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:23:35.262258 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c815aa48-cdeb-45a9-86ae-04cdaf994239 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:35.262715 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c815aa48-cdeb-45a9-86ae-04cdaf994239 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:35.262866 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c815aa48-cdeb-45a9-86ae-04cdaf994239 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:35.279999 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-c815aa48-cdeb-45a9-86ae-04cdaf994239 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:23:35.281193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-c815aa48-cdeb-45a9-86ae-04cdaf994239 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:23:35.350127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 636/1275] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:23:35 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:23:35.374087 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a73cd0-ac55-445e-aef3-391cf37c93cf None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:35.476374 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a73cd0-ac55-445e-aef3-391cf37c93cf None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:35.493516 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:35.511168 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed18015-5f73-44d9-b93a-d6fb2e5b90eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:35.511892 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed18015-5f73-44d9-b93a-d6fb2e5b90eb None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:35.512178 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed18015-5f73-44d9-b93a-d6fb2e5b90eb None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:35.512455 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed18015-5f73-44d9-b93a-d6fb2e5b90eb None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:35.522881 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed18015-5f73-44d9-b93a-d6fb2e5b90eb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 12 17:23:35.530604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-2ed18015-5f73-44d9-b93a-d6fb2e5b90eb None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:23:35.530604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:35.530604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:23:35.530604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:23:35.530604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:23:35.530604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:23:35.530604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:23:35.530604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:23:35.530604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:23:35.530604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:23:35.530604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:23:35.530604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:23:35.530604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 12 17:23:35.530604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 12 17:23:35.530604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 12 17:23:35.530604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 12 17:23:35.532352 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 12 17:23:35.532352 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 12 17:23:35.532352 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 12 17:23:35.532352 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 12 17:23:35.532352 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:23:35.532352 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:23:35.532352 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:23:35.532352 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:23:35.532352 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:23:35.532352 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:23:35.532352 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:23:35.532352 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:23:35.532352 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:23:35.532352 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:23:35.532352 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:23:35.532352 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:23:35.533643 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:23:35.533643 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:23:35.533643 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:23:35.533643 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:23:35.533643 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 12 17:23:35.533643 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 12 17:23:35.533643 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 12 17:23:35.533643 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 12 17:23:35.533643 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 12 17:23:35.533643 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 12 17:23:35.533643 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 12 17:23:35.533643 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:23:35.533643 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 640/1276] 10.222.0.16 () {56 vars in 1096 bytes} [Tue Jul 12 17:23:35 2022] GET /identity/v3/roles/admin => generated 83 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:23:35.538608 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c815aa48-cdeb-45a9-86ae-04cdaf994239 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:35.627394 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:35.645395 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c815aa48-cdeb-45a9-86ae-04cdaf994239 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:35.680526 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c170cb-f743-449d-af14-55de44d9bff2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:35.681193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c170cb-f743-449d-af14-55de44d9bff2 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:35.681476 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c170cb-f743-449d-af14-55de44d9bff2 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:35.681757 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c170cb-f743-449d-af14-55de44d9bff2 None admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:35.682530 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c170cb-f743-449d-af14-55de44d9bff2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:35.683744 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c170cb-f743-449d-af14-55de44d9bff2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:35.684183 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c170cb-f743-449d-af14-55de44d9bff2 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:35.692337 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 637/1277] 10.222.0.16 () {56 vars in 1099 bytes} [Tue Jul 12 17:23:35 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:23:35.698110 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed18015-5f73-44d9-b93a-d6fb2e5b90eb None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:35.801695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed18015-5f73-44d9-b93a-d6fb2e5b90eb None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:35.835695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29123e66-1e23-4a11-9e07-25959b1ede86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:35.836739 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29123e66-1e23-4a11-9e07-25959b1ede86 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:35.837023 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29123e66-1e23-4a11-9e07-25959b1ede86 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:35.837306 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29123e66-1e23-4a11-9e07-25959b1ede86 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:35.848836 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29123e66-1e23-4a11-9e07-25959b1ede86 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:35.850302 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29123e66-1e23-4a11-9e07-25959b1ede86 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:35.867820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-29123e66-1e23-4a11-9e07-25959b1ede86 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:23:35.867820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:35.867820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:23:35.867820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:23:35.867820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:23:35.867820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:23:35.867820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:23:35.867820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:23:35.867820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:23:35.867820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:23:35.867820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:23:35.867820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:23:35.867820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:23:35.867820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:23:35.867820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:23:35.869071 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:23:35.869071 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:23:35.869071 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:23:35.869071 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:23:35.869071 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:23:35.869071 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:23:35.869071 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:23:35.869071 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:23:35.869071 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:23:35.869071 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:23:35.869071 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:23:35.869071 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:23:35.869071 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:23:35.869071 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:23:35.869071 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:23:35.869071 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:23:35.869071 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:23:35.870102 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:23:35.870102 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:23:35.870102 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:23:35.870102 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:23:35.870102 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 641/1278] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:23:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 12 17:23:35.875807 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c170cb-f743-449d-af14-55de44d9bff2 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:35.982938 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c170cb-f743-449d-af14-55de44d9bff2 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:36.017200 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5287ec1-fc09-4441-bab6-77b5ad9a3fe0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:36.017864 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5287ec1-fc09-4441-bab6-77b5ad9a3fe0 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:36.018231 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5287ec1-fc09-4441-bab6-77b5ad9a3fe0 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:36.018461 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5287ec1-fc09-4441-bab6-77b5ad9a3fe0 None admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:36.019190 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5287ec1-fc09-4441-bab6-77b5ad9a3fe0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:36.019504 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5287ec1-fc09-4441-bab6-77b5ad9a3fe0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:36.020915 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5287ec1-fc09-4441-bab6-77b5ad9a3fe0 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:36.033014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 638/1279] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:23:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:23:36.038970 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29123e66-1e23-4a11-9e07-25959b1ede86 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:36.070774 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:36.142137 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29123e66-1e23-4a11-9e07-25959b1ede86 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:36.177259 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7408ef7-04bc-48c9-8b76-088fa2c518b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:36.178064 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7408ef7-04bc-48c9-8b76-088fa2c518b8 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:36.178355 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7408ef7-04bc-48c9-8b76-088fa2c518b8 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:36.178669 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7408ef7-04bc-48c9-8b76-088fa2c518b8 None admin] PATH_INFO: `/v3/users/placement` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:36.186259 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7408ef7-04bc-48c9-8b76-088fa2c518b8 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:36.188084 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7408ef7-04bc-48c9-8b76-088fa2c518b8 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:36.202689 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-f7408ef7-04bc-48c9-8b76-088fa2c518b8 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 12 17:23:36.202689 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:36.202689 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:23:36.202689 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:23:36.202689 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:23:36.202689 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:23:36.202689 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:23:36.202689 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:23:36.202689 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:23:36.202689 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:23:36.202689 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:23:36.202689 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:23:36.202689 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 12 17:23:36.202689 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 12 17:23:36.202689 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 12 17:23:36.202689 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 12 17:23:36.203850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:23:36.203850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:23:36.203850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 12 17:23:36.203850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:23:36.203850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 12 17:23:36.203850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 12 17:23:36.203850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:23:36.203850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:23:36.203850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:23:36.203850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:23:36.203850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:23:36.203850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:23:36.203850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:23:36.203850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:23:36.203850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:23:36.203850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:23:36.203850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:23:36.203850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:23:36.204860 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:23:36.204860 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:23:36.204860 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 12 17:23:36.204860 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 12 17:23:36.204860 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 12 17:23:36.204860 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 12 17:23:36.204860 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 12 17:23:36.204860 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 12 17:23:36.204860 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 12 17:23:36.204860 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:23:36.204860 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 642/1280] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:23:36 2022] GET /identity/v3/users/placement => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:23:36.209919 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5287ec1-fc09-4441-bab6-77b5ad9a3fe0 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:36.214048 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:36.313244 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5287ec1-fc09-4441-bab6-77b5ad9a3fe0 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:36.347814 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4d450d-51d1-425d-a276-0b69b52e888d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:36.348423 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4d450d-51d1-425d-a276-0b69b52e888d None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:36.348715 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4d450d-51d1-425d-a276-0b69b52e888d None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:36.348998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4d450d-51d1-425d-a276-0b69b52e888d None admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:36.349905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4d450d-51d1-425d-a276-0b69b52e888d None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:36.350221 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4d450d-51d1-425d-a276-0b69b52e888d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:36.351690 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4d450d-51d1-425d-a276-0b69b52e888d None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:36.369810 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 639/1281] 10.222.0.16 () {56 vars in 1143 bytes} [Tue Jul 12 17:23:36 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:23:36.375639 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7408ef7-04bc-48c9-8b76-088fa2c518b8 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:36.467236 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:36.477281 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7408ef7-04bc-48c9-8b76-088fa2c518b8 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:36.510266 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e8be67-3331-4721-82b4-44c16fd87a58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:36.510940 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e8be67-3331-4721-82b4-44c16fd87a58 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:36.511232 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e8be67-3331-4721-82b4-44c16fd87a58 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:36.511513 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e8be67-3331-4721-82b4-44c16fd87a58 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:36.580948 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e8be67-3331-4721-82b4-44c16fd87a58 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:36.582589 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e8be67-3331-4721-82b4-44c16fd87a58 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:36.676155 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:36.685972 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 643/1282] 10.222.0.16 () {58 vars in 1282 bytes} [Tue Jul 12 17:23:36 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 12 17:23:36.691999 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4d450d-51d1-425d-a276-0b69b52e888d None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:36.760424 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:36.800839 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4d450d-51d1-425d-a276-0b69b52e888d None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:36.833947 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7116db3-878d-4e2f-a4f2-74ddac3ce4ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:36.834816 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7116db3-878d-4e2f-a4f2-74ddac3ce4ed None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:36.835271 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7116db3-878d-4e2f-a4f2-74ddac3ce4ed None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:36.835559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7116db3-878d-4e2f-a4f2-74ddac3ce4ed None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:36.847400 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7116db3-878d-4e2f-a4f2-74ddac3ce4ed None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:36.848859 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7116db3-878d-4e2f-a4f2-74ddac3ce4ed None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:36.866388 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-e7116db3-878d-4e2f-a4f2-74ddac3ce4ed None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:23:36.866388 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:36.866388 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:23:36.866388 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:23:36.866388 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:23:36.866388 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:23:36.866388 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:23:36.866388 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:23:36.866388 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:23:36.866388 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:23:36.866388 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:23:36.866388 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:23:36.866388 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 12 17:23:36.866388 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 12 17:23:36.866388 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 12 17:23:36.867520 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 12 17:23:36.867520 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:23:36.867520 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:23:36.867520 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:23:36.867520 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:23:36.867520 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:23:36.867520 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:23:36.867520 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:23:36.867520 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:23:36.867520 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:23:36.867520 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:23:36.867520 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:23:36.867520 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:23:36.867520 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:23:36.867520 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:23:36.867520 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:23:36.867520 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:23:36.868657 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 12 17:23:36.868657 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 12 17:23:36.868657 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 12 17:23:36.868657 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:23:36.868657 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 640/1283] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:23:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:23:36.873586 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e8be67-3331-4721-82b4-44c16fd87a58 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:36.974692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e8be67-3331-4721-82b4-44c16fd87a58 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:37.008724 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ecd5e8-c34d-4886-9d5b-1fd8dbbd452a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:37.009541 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ecd5e8-c34d-4886-9d5b-1fd8dbbd452a None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:37.009894 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ecd5e8-c34d-4886-9d5b-1fd8dbbd452a None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:37.010178 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ecd5e8-c34d-4886-9d5b-1fd8dbbd452a None admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:37.010938 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ecd5e8-c34d-4886-9d5b-1fd8dbbd452a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:37.011251 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ecd5e8-c34d-4886-9d5b-1fd8dbbd452a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:37.012714 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ecd5e8-c34d-4886-9d5b-1fd8dbbd452a None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:37.024177 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 644/1284] 10.222.0.16 () {56 vars in 1109 bytes} [Tue Jul 12 17:23:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:23:37.029909 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7116db3-878d-4e2f-a4f2-74ddac3ce4ed None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:37.129788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7116db3-878d-4e2f-a4f2-74ddac3ce4ed None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:37.143458 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:37.163233 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035bb5c2-07a5-40df-a2ab-0123c0b7ac6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:37.164021 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035bb5c2-07a5-40df-a2ab-0123c0b7ac6e None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:37.164309 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035bb5c2-07a5-40df-a2ab-0123c0b7ac6e None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:37.164587 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035bb5c2-07a5-40df-a2ab-0123c0b7ac6e None admin] PATH_INFO: `/v3/projects/service` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:37.171081 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035bb5c2-07a5-40df-a2ab-0123c0b7ac6e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:37.173290 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035bb5c2-07a5-40df-a2ab-0123c0b7ac6e None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:37.187221 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application [None req-035bb5c2-07a5-40df-a2ab-0123c0b7ac6e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:23:37.187221 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:37.187221 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:23:37.187221 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:23:37.187221 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:23:37.187221 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:23:37.187221 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:23:37.187221 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:23:37.187221 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:23:37.187221 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:23:37.187221 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:23:37.187221 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:23:37.187221 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 12 17:23:37.187221 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 12 17:23:37.187221 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 12 17:23:37.188736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 12 17:23:37.188736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:23:37.188736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:23:37.188736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:23:37.188736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:23:37.188736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:23:37.188736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:23:37.188736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:23:37.188736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application with Lock( Jul 12 17:23:37.188736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:23:37.188736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:23:37.188736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:23:37.188736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:23:37.188736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:23:37.188736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:23:37.188736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:23:37.188736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:23:37.189994 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 12 17:23:37.189994 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 12 17:23:37.189994 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 12 17:23:37.189994 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 12 17:23:37.189994 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 12 17:23:37.189994 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 12 17:23:37.189994 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 12 17:23:37.189994 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: ERROR keystone.server.flask.application Jul 12 17:23:37.189994 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 641/1285] 10.222.0.16 () {56 vars in 1111 bytes} [Tue Jul 12 17:23:37 2022] GET /identity/v3/projects/service => generated 88 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:23:37.194407 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ecd5e8-c34d-4886-9d5b-1fd8dbbd452a None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:37.245156 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:37.293940 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ecd5e8-c34d-4886-9d5b-1fd8dbbd452a None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:37.328746 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3fca7f-4de7-4d77-a334-6906a9667307 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:37.333561 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3fca7f-4de7-4d77-a334-6906a9667307 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:37.333561 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3fca7f-4de7-4d77-a334-6906a9667307 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:37.333561 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3fca7f-4de7-4d77-a334-6906a9667307 None admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:37.333561 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3fca7f-4de7-4d77-a334-6906a9667307 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:37.333561 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3fca7f-4de7-4d77-a334-6906a9667307 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:37.333561 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3fca7f-4de7-4d77-a334-6906a9667307 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:37.345754 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 645/1286] 10.222.0.16 () {56 vars in 1148 bytes} [Tue Jul 12 17:23:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:23:37.351912 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035bb5c2-07a5-40df-a2ab-0123c0b7ac6e None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:37.454892 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035bb5c2-07a5-40df-a2ab-0123c0b7ac6e None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:37.493253 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dec977d-95ca-4a59-a152-7f6537efcb40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:37.493951 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dec977d-95ca-4a59-a152-7f6537efcb40 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:37.494274 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dec977d-95ca-4a59-a152-7f6537efcb40 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:37.494603 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dec977d-95ca-4a59-a152-7f6537efcb40 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:37.495497 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dec977d-95ca-4a59-a152-7f6537efcb40 None admin] RBAC: Adding query filter params (role.id=eb05bf286c894ea9bf44a6c023a4323b, scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2, user.id=f623be28c83247c78d3ea6873c347fe0) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:37.495769 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dec977d-95ca-4a59-a152-7f6537efcb40 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:37.497210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dec977d-95ca-4a59-a152-7f6537efcb40 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:37.504453 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 642/1287] 10.222.0.16 () {56 vars in 1374 bytes} [Tue Jul 12 17:23:37 2022] GET /identity/v3/role_assignments?user.id=f623be28c83247c78d3ea6873c347fe0&scope.project.id=fd4ed2fe037c4fb7ba49c724eb9414c2&role.id=eb05bf286c894ea9bf44a6c023a4323b => generated 608 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:23:39.609299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d4c06b-da08-48e1-9cad-e6f0f1cd4ea5 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:39.609856 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d4c06b-da08-48e1-9cad-e6f0f1cd4ea5 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:39.609856 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d4c06b-da08-48e1-9cad-e6f0f1cd4ea5 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:39.611403 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 646/1288] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:23:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:23:39.616828 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1e0e00-90d3-4c8b-95a0-a8fa92cf5c40 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:39.617238 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1e0e00-90d3-4c8b-95a0-a8fa92cf5c40 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:39.617414 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1e0e00-90d3-4c8b-95a0-a8fa92cf5c40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:39.634542 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-ca1e0e00-90d3-4c8b-95a0-a8fa92cf5c40 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:23:39.635812 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-ca1e0e00-90d3-4c8b-95a0-a8fa92cf5c40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:23:39.666422 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:39.703222 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 643/1289] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:23:39 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:23:39.727576 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d4c06b-da08-48e1-9cad-e6f0f1cd4ea5 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:39.829818 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d4c06b-da08-48e1-9cad-e6f0f1cd4ea5 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:39.840588 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:39.863974 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78ace28-69e1-4f69-91df-547de1b5257c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:39.864774 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78ace28-69e1-4f69-91df-547de1b5257c None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:39.865060 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78ace28-69e1-4f69-91df-547de1b5257c None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:39.865340 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78ace28-69e1-4f69-91df-547de1b5257c None admin] PATH_INFO: `/v3/services/placement` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:39.865994 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a78ace28-69e1-4f69-91df-547de1b5257c None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:39.867387 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a78ace28-69e1-4f69-91df-547de1b5257c None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:39.879240 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-a78ace28-69e1-4f69-91df-547de1b5257c None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Jul 12 17:23:39.879240 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:39.879240 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 12 17:23:39.879240 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 12 17:23:39.879240 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 12 17:23:39.879240 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 12 17:23:39.879240 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 12 17:23:39.879240 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:23:39.879240 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 12 17:23:39.879240 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:23:39.879240 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 12 17:23:39.879240 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:23:39.879240 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:39.879240 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:23:39.879240 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:23:39.879240 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:23:39.879240 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:23:39.880298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:23:39.880298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:23:39.880298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:23:39.880298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:23:39.880298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:23:39.880298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:23:39.880298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 12 17:23:39.880298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 12 17:23:39.880298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 12 17:23:39.880298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 12 17:23:39.880298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:23:39.880298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:23:39.880298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:23:39.880298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:23:39.880298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:23:39.880298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:23:39.881245 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:23:39.881245 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:23:39.881245 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:23:39.881245 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:23:39.881245 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:23:39.881245 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:23:39.881245 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:23:39.881245 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:23:39.881245 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:23:39.881245 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:23:39.881245 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 12 17:23:39.881245 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:23:39.881245 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 12 17:23:39.881245 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:23:39.881245 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 647/1290] 10.222.0.16 () {56 vars in 1117 bytes} [Tue Jul 12 17:23:39 2022] GET /identity/v3/services/placement => generated 90 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:23:39.886681 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1e0e00-90d3-4c8b-95a0-a8fa92cf5c40 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:39.986302 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1e0e00-90d3-4c8b-95a0-a8fa92cf5c40 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:40.020944 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f8309f-ae00-4b41-9f8e-1d4c84cd50a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:40.021616 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f8309f-ae00-4b41-9f8e-1d4c84cd50a4 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:40.021908 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f8309f-ae00-4b41-9f8e-1d4c84cd50a4 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:40.022195 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f8309f-ae00-4b41-9f8e-1d4c84cd50a4 None admin] PATH_INFO: `/v3/services` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:40.022951 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65f8309f-ae00-4b41-9f8e-1d4c84cd50a4 None admin] RBAC: Adding query filter params (name=placement) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:40.023267 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65f8309f-ae00-4b41-9f8e-1d4c84cd50a4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:40.024698 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65f8309f-ae00-4b41-9f8e-1d4c84cd50a4 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:40.029779 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 644/1291] 10.222.0.16 () {56 vars in 1116 bytes} [Tue Jul 12 17:23:39 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:23:40.035716 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78ace28-69e1-4f69-91df-547de1b5257c None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:40.136809 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78ace28-69e1-4f69-91df-547de1b5257c None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:40.171075 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2eda0aa-78c3-4475-bbd8-edb601f8ee05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:40.171737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2eda0aa-78c3-4475-bbd8-edb601f8ee05 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:40.172025 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2eda0aa-78c3-4475-bbd8-edb601f8ee05 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:40.172366 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2eda0aa-78c3-4475-bbd8-edb601f8ee05 None admin] PATH_INFO: `/v3/services` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:40.173128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2eda0aa-78c3-4475-bbd8-edb601f8ee05 None admin] RBAC: Adding query filter params (type=placement) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:40.173444 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2eda0aa-78c3-4475-bbd8-edb601f8ee05 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:40.174844 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2eda0aa-78c3-4475-bbd8-edb601f8ee05 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:40.179335 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 648/1292] 10.222.0.16 () {56 vars in 1116 bytes} [Tue Jul 12 17:23:40 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:23:42.231859 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2849f05-d0a4-4605-9983-87b9f431a4fa None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:42.232461 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2849f05-d0a4-4605-9983-87b9f431a4fa None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:42.232461 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2849f05-d0a4-4605-9983-87b9f431a4fa None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:42.233824 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 645/1293] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:23:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:23:42.239604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f087d4-3388-4692-a7a4-cf1a9042b4bc None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:42.240016 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f087d4-3388-4692-a7a4-cf1a9042b4bc None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:42.240199 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f087d4-3388-4692-a7a4-cf1a9042b4bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:42.257376 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-54f087d4-3388-4692-a7a4-cf1a9042b4bc None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:23:42.263989 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-54f087d4-3388-4692-a7a4-cf1a9042b4bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:23:42.289545 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:42.326763 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 649/1294] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:23:42 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:23:42.350831 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2849f05-d0a4-4605-9983-87b9f431a4fa None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:42.382936 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:42.457333 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2849f05-d0a4-4605-9983-87b9f431a4fa None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:42.493755 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41f717e6-3b35-4a13-97c4-4c877bc0b35f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:42.494347 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f717e6-3b35-4a13-97c4-4c877bc0b35f None admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:42.494663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f717e6-3b35-4a13-97c4-4c877bc0b35f None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:42.494950 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41f717e6-3b35-4a13-97c4-4c877bc0b35f None admin] PATH_INFO: `/v3/services` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:42.495675 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f717e6-3b35-4a13-97c4-4c877bc0b35f None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:42.497038 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41f717e6-3b35-4a13-97c4-4c877bc0b35f None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:42.603459 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 646/1295] 10.222.0.16 () {60 vars in 1141 bytes} [Tue Jul 12 17:23:42 2022] POST /identity/v3/services => generated 246 bytes in 253 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:23:44.672909 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64da210d-7909-4e58-8170-3563267a3775 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:44.673434 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64da210d-7909-4e58-8170-3563267a3775 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:44.673551 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64da210d-7909-4e58-8170-3563267a3775 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:44.675467 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 650/1296] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:23:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:23:44.680751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97197a5e-29b8-4d58-b799-5da9aa1cb25a None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:44.681172 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97197a5e-29b8-4d58-b799-5da9aa1cb25a None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:44.681319 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97197a5e-29b8-4d58-b799-5da9aa1cb25a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:44.698223 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-97197a5e-29b8-4d58-b799-5da9aa1cb25a None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:23:44.699954 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-97197a5e-29b8-4d58-b799-5da9aa1cb25a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:23:44.772843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 647/1297] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:23:44 2022] POST /identity/v3/auth/tokens => generated 2211 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:23:44.796237 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64da210d-7909-4e58-8170-3563267a3775 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:44.900124 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64da210d-7909-4e58-8170-3563267a3775 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:44.934226 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652e6f19-7ff9-4727-81d0-e4a87187992a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:44.935042 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652e6f19-7ff9-4727-81d0-e4a87187992a None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:44.935331 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652e6f19-7ff9-4727-81d0-e4a87187992a None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:44.935612 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652e6f19-7ff9-4727-81d0-e4a87187992a None admin] PATH_INFO: `/v3/services/placement` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:44.936261 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652e6f19-7ff9-4727-81d0-e4a87187992a None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:44.937623 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652e6f19-7ff9-4727-81d0-e4a87187992a None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:44.950243 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-652e6f19-7ff9-4727-81d0-e4a87187992a None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Jul 12 17:23:44.950243 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:44.950243 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 12 17:23:44.950243 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 12 17:23:44.950243 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 12 17:23:44.950243 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 12 17:23:44.950243 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 12 17:23:44.950243 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:23:44.950243 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 12 17:23:44.950243 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:23:44.950243 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 12 17:23:44.950243 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:23:44.950243 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:44.950243 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:23:44.950243 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:23:44.950243 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:23:44.950243 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:23:44.951651 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:23:44.951651 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:23:44.951651 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:23:44.951651 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:23:44.951651 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:23:44.951651 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:23:44.951651 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 12 17:23:44.951651 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 12 17:23:44.951651 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 12 17:23:44.951651 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 12 17:23:44.951651 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:23:44.951651 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:23:44.951651 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:23:44.951651 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:23:44.951651 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:23:44.951651 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:23:44.952957 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:23:44.952957 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:23:44.952957 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:23:44.952957 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:23:44.952957 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:23:44.952957 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:23:44.952957 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:23:44.952957 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:23:44.952957 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:23:44.952957 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:23:44.952957 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 12 17:23:44.952957 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:23:44.952957 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 12 17:23:44.952957 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:23:44.952957 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 651/1298] 10.222.0.16 () {56 vars in 1117 bytes} [Tue Jul 12 17:23:44 2022] GET /identity/v3/services/placement => generated 90 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:23:44.957606 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97197a5e-29b8-4d58-b799-5da9aa1cb25a None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:44.976279 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:45.055670 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97197a5e-29b8-4d58-b799-5da9aa1cb25a None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:45.090367 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c83520-ca02-40ac-af25-6fc6c6bca388 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:45.091066 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c83520-ca02-40ac-af25-6fc6c6bca388 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:45.091357 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c83520-ca02-40ac-af25-6fc6c6bca388 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:45.091640 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c83520-ca02-40ac-af25-6fc6c6bca388 None admin] PATH_INFO: `/v3/services` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:45.092354 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3c83520-ca02-40ac-af25-6fc6c6bca388 None admin] RBAC: Adding query filter params (name=placement) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:45.092666 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3c83520-ca02-40ac-af25-6fc6c6bca388 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:45.094093 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3c83520-ca02-40ac-af25-6fc6c6bca388 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:45.099361 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 648/1299] 10.222.0.16 () {56 vars in 1116 bytes} [Tue Jul 12 17:23:44 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:23:45.105425 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652e6f19-7ff9-4727-81d0-e4a87187992a None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:45.137379 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:45.203872 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652e6f19-7ff9-4727-81d0-e4a87187992a None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:45.237096 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee5d66c-ecb8-43a7-9dba-3877c5bf19c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:45.237750 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee5d66c-ecb8-43a7-9dba-3877c5bf19c6 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:45.238034 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee5d66c-ecb8-43a7-9dba-3877c5bf19c6 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:45.238307 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee5d66c-ecb8-43a7-9dba-3877c5bf19c6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:45.239122 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee5d66c-ecb8-43a7-9dba-3877c5bf19c6 None admin] RBAC: Adding query filter params (interface=public, service_id=96c2ba8904c8460086add1d3361c06e4, region_id=RegionOne) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:45.239434 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee5d66c-ecb8-43a7-9dba-3877c5bf19c6 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:45.240793 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee5d66c-ecb8-43a7-9dba-3877c5bf19c6 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:45.245965 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 652/1300] 10.222.0.16 () {56 vars in 1251 bytes} [Tue Jul 12 17:23:45 2022] GET /identity/v3/endpoints?service_id=96c2ba8904c8460086add1d3361c06e4&interface=public®ion_id=RegionOne => generated 194 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:23:45.251784 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c83520-ca02-40ac-af25-6fc6c6bca388 None admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:45.334451 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:45.353349 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c83520-ca02-40ac-af25-6fc6c6bca388 None admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:45.388848 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9247c12c-9293-4aec-bb47-c684192280f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:45.389516 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9247c12c-9293-4aec-bb47-c684192280f0 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:45.389809 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9247c12c-9293-4aec-bb47-c684192280f0 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:45.390098 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9247c12c-9293-4aec-bb47-c684192280f0 None admin] PATH_INFO: `/v3/services` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:45.390864 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9247c12c-9293-4aec-bb47-c684192280f0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:45.392245 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9247c12c-9293-4aec-bb47-c684192280f0 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:45.399205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 649/1301] 10.222.0.16 () {56 vars in 1087 bytes} [Tue Jul 12 17:23:45 2022] GET /identity/v3/services => generated 2191 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:23:47.471662 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67cf9cc5-3684-4802-85a7-6ce354ba3166 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:47.472266 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67cf9cc5-3684-4802-85a7-6ce354ba3166 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:47.472266 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67cf9cc5-3684-4802-85a7-6ce354ba3166 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:47.473590 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 653/1302] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:23:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:23:47.479084 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c742815a-d409-4585-bde7-9964ebff42aa None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:47.479532 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c742815a-d409-4585-bde7-9964ebff42aa None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:47.481008 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c742815a-d409-4585-bde7-9964ebff42aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:47.496627 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-c742815a-d409-4585-bde7-9964ebff42aa None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:23:47.497820 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-c742815a-d409-4585-bde7-9964ebff42aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:23:47.563688 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 650/1303] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:23:47 2022] POST /identity/v3/auth/tokens => generated 2211 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 12 17:23:47.588013 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67cf9cc5-3684-4802-85a7-6ce354ba3166 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:47.684687 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:47.692143 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67cf9cc5-3684-4802-85a7-6ce354ba3166 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:47.727300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d124ae14-ef12-4559-a7db-100c2cbf5f00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:47.728022 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d124ae14-ef12-4559-a7db-100c2cbf5f00 None admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:47.728305 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d124ae14-ef12-4559-a7db-100c2cbf5f00 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:47.728581 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d124ae14-ef12-4559-a7db-100c2cbf5f00 None admin] PATH_INFO: `/v3/services/placement` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:47.729230 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d124ae14-ef12-4559-a7db-100c2cbf5f00 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:47.730612 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d124ae14-ef12-4559-a7db-100c2cbf5f00 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:47.742532 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application [None req-d124ae14-ef12-4559-a7db-100c2cbf5f00 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Jul 12 17:23:47.742532 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:47.742532 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 12 17:23:47.742532 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 12 17:23:47.742532 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 12 17:23:47.742532 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 12 17:23:47.742532 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 12 17:23:47.742532 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:23:47.742532 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 12 17:23:47.742532 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:23:47.742532 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 12 17:23:47.742532 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:23:47.742532 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 12 17:23:47.742532 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 12 17:23:47.742532 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 12 17:23:47.742532 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 12 17:23:47.742532 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 12 17:23:47.743837 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 12 17:23:47.743837 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 12 17:23:47.743837 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 12 17:23:47.743837 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 12 17:23:47.743837 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 12 17:23:47.743837 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 12 17:23:47.743837 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 12 17:23:47.743837 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 12 17:23:47.743837 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 12 17:23:47.743837 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 12 17:23:47.743837 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 12 17:23:47.743837 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 12 17:23:47.743837 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 12 17:23:47.743837 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 12 17:23:47.743837 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 12 17:23:47.743837 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.get_or_create( Jul 12 17:23:47.744867 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 12 17:23:47.744867 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application with Lock( Jul 12 17:23:47.744867 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 12 17:23:47.744867 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self._enter() Jul 12 17:23:47.744867 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 12 17:23:47.744867 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 12 17:23:47.744867 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 12 17:23:47.744867 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application return self.creator() Jul 12 17:23:47.744867 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 12 17:23:47.744867 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application created_value = creator( Jul 12 17:23:47.744867 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 12 17:23:47.744867 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 12 17:23:47.744867 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 12 17:23:47.744867 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: ERROR keystone.server.flask.application Jul 12 17:23:47.744867 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 654/1304] 10.222.0.16 () {56 vars in 1117 bytes} [Tue Jul 12 17:23:47 2022] GET /identity/v3/services/placement => generated 90 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 12 17:23:47.749990 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c742815a-d409-4585-bde7-9964ebff42aa None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:47.828270 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:47.851291 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c742815a-d409-4585-bde7-9964ebff42aa None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:47.885968 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04675d5d-59e5-48b3-ac7d-ae9f391aa0b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:47.886663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04675d5d-59e5-48b3-ac7d-ae9f391aa0b3 None admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:47.886957 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04675d5d-59e5-48b3-ac7d-ae9f391aa0b3 None admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:47.887244 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04675d5d-59e5-48b3-ac7d-ae9f391aa0b3 None admin] PATH_INFO: `/v3/services` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:47.887958 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04675d5d-59e5-48b3-ac7d-ae9f391aa0b3 None admin] RBAC: Adding query filter params (name=placement) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:23:47.888273 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04675d5d-59e5-48b3-ac7d-ae9f391aa0b3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:47.889695 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04675d5d-59e5-48b3-ac7d-ae9f391aa0b3 None admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:47.895058 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 651/1305] 10.222.0.16 () {56 vars in 1116 bytes} [Tue Jul 12 17:23:47 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:23:47.900894 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d124ae14-ef12-4559-a7db-100c2cbf5f00 None admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:23:48.004469 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d124ae14-ef12-4559-a7db-100c2cbf5f00 None admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:23:48.037225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afe4f0bc-5476-4524-984e-b2245d0d8280 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:23:48.037878 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe4f0bc-5476-4524-984e-b2245d0d8280 None admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:23:48.038168 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe4f0bc-5476-4524-984e-b2245d0d8280 None admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:23:48.038468 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afe4f0bc-5476-4524-984e-b2245d0d8280 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:23:48.039205 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe4f0bc-5476-4524-984e-b2245d0d8280 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:23:48.040568 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afe4f0bc-5476-4524-984e-b2245d0d8280 None admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:23:48.130808 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 655/1306] 10.222.0.16 () {60 vars in 1144 bytes} [Tue Jul 12 17:23:47 2022] POST /identity/v3/endpoints => generated 331 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:23:57.917932 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=30 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:23:58.125220 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:30:12.818538 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf020c5-7469-4944-b47f-ac6640d6a330 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:12.819226 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf020c5-7469-4944-b47f-ac6640d6a330 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:12.819226 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf020c5-7469-4944-b47f-ac6640d6a330 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:12.820513 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 652/1307] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:30:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:30:12.826011 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ec5ec0-7832-4a76-94c4-7cace9291e28 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:12.826537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ec5ec0-7832-4a76-94c4-7cace9291e28 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:12.826649 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ec5ec0-7832-4a76-94c4-7cace9291e28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:13.148385 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-30ec5ec0-7832-4a76-94c4-7cace9291e28 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:30:13.149628 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-30ec5ec0-7832-4a76-94c4-7cace9291e28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:13.319048 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 656/1308] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:30:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 494 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:30:13.355715 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0540d87-3cbc-4d16-a82e-99828a0892eb None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:13.356265 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0540d87-3cbc-4d16-a82e-99828a0892eb None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:13.356707 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0540d87-3cbc-4d16-a82e-99828a0892eb None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:13.358720 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 653/1309] 10.222.0.16 () {54 vars in 964 bytes} [Tue Jul 12 17:30:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:30:13.365157 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2505760b-dad3-43ba-9c88-0112a756ef72 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:13.365447 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2505760b-dad3-43ba-9c88-0112a756ef72 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:13.365915 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2505760b-dad3-43ba-9c88-0112a756ef72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:13.460356 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-2505760b-dad3-43ba-9c88-0112a756ef72 None None] MFA Rules not processed for user `bf5bf02ae13d4d86a1b157369ef07a00`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:30:13.461846 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-2505760b-dad3-43ba-9c88-0112a756ef72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:13.489789 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:30:13.550827 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 657/1310] 10.222.0.16 () {58 vars in 1061 bytes} [Tue Jul 12 17:30:13 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:30:13.559937 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0540d87-3cbc-4d16-a82e-99828a0892eb None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:30:13.561843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-a0540d87-3cbc-4d16-a82e-99828a0892eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:13.784079 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0540d87-3cbc-4d16-a82e-99828a0892eb None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:30:13.910271 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8642099f-85e6-440f-8691-948bafbedf8d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:30:13.911132 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8642099f-85e6-440f-8691-948bafbedf8d service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:13.911529 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8642099f-85e6-440f-8691-948bafbedf8d service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:13.911924 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8642099f-85e6-440f-8691-948bafbedf8d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:13.914342 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-8642099f-85e6-440f-8691-948bafbedf8d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:13.976770 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:30:14.051695 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8642099f-85e6-440f-8691-948bafbedf8d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:30:14.053536 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8642099f-85e6-440f-8691-948bafbedf8d service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:30:14.075754 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 654/1311] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:30:13 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 516 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:30:16.480757 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5524af58-8688-48bb-ba35-73dc58d6830f None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:16.481338 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5524af58-8688-48bb-ba35-73dc58d6830f None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:16.481338 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5524af58-8688-48bb-ba35-73dc58d6830f None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:16.482873 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 658/1312] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:30:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:30:16.488284 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4baa9748-82bb-49ad-ac84-d88d9af995c2 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:16.488754 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4baa9748-82bb-49ad-ac84-d88d9af995c2 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:16.488849 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4baa9748-82bb-49ad-ac84-d88d9af995c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:16.603103 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-4baa9748-82bb-49ad-ac84-d88d9af995c2 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:30:16.604405 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-4baa9748-82bb-49ad-ac84-d88d9af995c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:16.733574 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 655/1313] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:30:16 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 246 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:30:16.771722 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5524af58-8688-48bb-ba35-73dc58d6830f None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:30:16.774010 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-5524af58-8688-48bb-ba35-73dc58d6830f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:16.890759 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5524af58-8688-48bb-ba35-73dc58d6830f None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:30:16.972918 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:30:16.976860 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26561c6a-af99-4fb9-80d2-41ec9694b7f4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:30:16.977687 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26561c6a-af99-4fb9-80d2-41ec9694b7f4 service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:16.978050 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26561c6a-af99-4fb9-80d2-41ec9694b7f4 service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:16.978401 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26561c6a-af99-4fb9-80d2-41ec9694b7f4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:16.980172 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-26561c6a-af99-4fb9-80d2-41ec9694b7f4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:17.106935 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26561c6a-af99-4fb9-80d2-41ec9694b7f4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:30:17.108401 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26561c6a-af99-4fb9-80d2-41ec9694b7f4 service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:30:17.130258 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 659/1314] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:30:16 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:30:19.704685 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2237ee9f-513d-415d-b6d9-8799d387e3d5 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:19.705232 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2237ee9f-513d-415d-b6d9-8799d387e3d5 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:19.705232 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2237ee9f-513d-415d-b6d9-8799d387e3d5 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:19.706661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 656/1315] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:30:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:30:19.712394 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaef2202-3fe6-4fd3-a65a-1bf2399a8cbf None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:19.712813 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaef2202-3fe6-4fd3-a65a-1bf2399a8cbf None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:19.712951 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaef2202-3fe6-4fd3-a65a-1bf2399a8cbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:19.779585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-aaef2202-3fe6-4fd3-a65a-1bf2399a8cbf None None] MFA Rules not processed for user `dbc90e3cb55f431e91cd23da7b8958f2`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:30:19.780813 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-aaef2202-3fe6-4fd3-a65a-1bf2399a8cbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:19.834079 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:30:19.878680 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 660/1316] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:30:19 2022] POST /identity/v3/auth/tokens => generated 3233 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:30:19.918627 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2237ee9f-513d-415d-b6d9-8799d387e3d5 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:30:19.920175 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-2237ee9f-513d-415d-b6d9-8799d387e3d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:19.930524 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:30:20.029667 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2237ee9f-513d-415d-b6d9-8799d387e3d5 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:30:20.105629 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a82603a-585a-425b-9dfd-46d3bcb7a9a3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:30:20.106227 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a82603a-585a-425b-9dfd-46d3bcb7a9a3 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:20.106617 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a82603a-585a-425b-9dfd-46d3bcb7a9a3 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:20.106916 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a82603a-585a-425b-9dfd-46d3bcb7a9a3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:20.108568 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-0a82603a-585a-425b-9dfd-46d3bcb7a9a3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:20.171286 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:30:20.219873 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a82603a-585a-425b-9dfd-46d3bcb7a9a3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:30:20.221579 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a82603a-585a-425b-9dfd-46d3bcb7a9a3 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:30:20.240932 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 657/1317] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:30:19 2022] GET /identity/v3/auth/tokens => generated 3233 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:30:23.201681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778c4c05-091c-45f3-a3b5-d3fe85a6c6b2 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:23.202250 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778c4c05-091c-45f3-a3b5-d3fe85a6c6b2 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:23.202366 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778c4c05-091c-45f3-a3b5-d3fe85a6c6b2 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:23.203774 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 661/1318] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:30:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:30:23.209576 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b64c1c-015c-4a9d-92ad-a37207d12c78 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:23.210090 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b64c1c-015c-4a9d-92ad-a37207d12c78 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:23.210188 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b64c1c-015c-4a9d-92ad-a37207d12c78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:23.281366 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-d0b64c1c-015c-4a9d-92ad-a37207d12c78 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:30:23.282535 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-d0b64c1c-015c-4a9d-92ad-a37207d12c78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:23.386084 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 658/1319] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:30:23 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:30:23.425104 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-778c4c05-091c-45f3-a3b5-d3fe85a6c6b2 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:30:23.426822 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-778c4c05-091c-45f3-a3b5-d3fe85a6c6b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:23.541686 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-778c4c05-091c-45f3-a3b5-d3fe85a6c6b2 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:30:23.621600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7fa04f4-ad19-4e53-8a36-def10c6d1480 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:30:23.622366 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fa04f4-ad19-4e53-8a36-def10c6d1480 service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:23.622736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fa04f4-ad19-4e53-8a36-def10c6d1480 service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:23.623024 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fa04f4-ad19-4e53-8a36-def10c6d1480 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:23.624899 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-a7fa04f4-ad19-4e53-8a36-def10c6d1480 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:23.633477 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:30:23.741567 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7fa04f4-ad19-4e53-8a36-def10c6d1480 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:30:23.743069 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7fa04f4-ad19-4e53-8a36-def10c6d1480 service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:30:23.762095 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 662/1320] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:30:23 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:30:26.880615 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d960674f-a727-4c44-abde-77e295fa2390 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:26.881188 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d960674f-a727-4c44-abde-77e295fa2390 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:26.881381 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d960674f-a727-4c44-abde-77e295fa2390 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:26.882871 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 659/1321] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:30:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:30:26.889973 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c5c024-2dea-45f0-8ba1-fbebe85327d8 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:26.890488 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c5c024-2dea-45f0-8ba1-fbebe85327d8 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:26.890583 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33c5c024-2dea-45f0-8ba1-fbebe85327d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:26.965854 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-33c5c024-2dea-45f0-8ba1-fbebe85327d8 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:30:26.967104 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-33c5c024-2dea-45f0-8ba1-fbebe85327d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:27.029552 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:30:27.072869 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 663/1322] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:30:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:30:27.109581 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d960674f-a727-4c44-abde-77e295fa2390 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:30:27.111177 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-d960674f-a727-4c44-abde-77e295fa2390 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:27.132126 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:30:27.234095 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d960674f-a727-4c44-abde-77e295fa2390 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:30:27.323906 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da0730b8-b0b1-4036-857a-66d9aca9a5f9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:30:27.324561 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0730b8-b0b1-4036-857a-66d9aca9a5f9 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:27.324846 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0730b8-b0b1-4036-857a-66d9aca9a5f9 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:27.325126 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da0730b8-b0b1-4036-857a-66d9aca9a5f9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:27.326871 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-da0730b8-b0b1-4036-857a-66d9aca9a5f9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:27.407906 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:30:27.448614 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0730b8-b0b1-4036-857a-66d9aca9a5f9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:30:27.450079 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da0730b8-b0b1-4036-857a-66d9aca9a5f9 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:30:27.471471 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 660/1323] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:30:27 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:30:29.729072 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4bc8df5-0368-44cd-9a46-5961ee84e73a None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:29.729618 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4bc8df5-0368-44cd-9a46-5961ee84e73a None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:29.729816 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4bc8df5-0368-44cd-9a46-5961ee84e73a None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:29.731459 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 664/1324] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:30:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:30:29.737586 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e358eb86-4705-4ffb-93bc-847491972182 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:29.738099 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e358eb86-4705-4ffb-93bc-847491972182 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:29.738198 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e358eb86-4705-4ffb-93bc-847491972182 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:29.818602 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-e358eb86-4705-4ffb-93bc-847491972182 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:30:29.819980 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-e358eb86-4705-4ffb-93bc-847491972182 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:29.960359 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 661/1325] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:30:29 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:30:29.997353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4bc8df5-0368-44cd-9a46-5961ee84e73a None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:30:29.999062 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-e4bc8df5-0368-44cd-9a46-5961ee84e73a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:30.148548 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4bc8df5-0368-44cd-9a46-5961ee84e73a None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:30:30.236258 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8706537-61d7-4b34-b99f-645c7b811ff3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:30:30.236776 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8706537-61d7-4b34-b99f-645c7b811ff3 service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:30.237067 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8706537-61d7-4b34-b99f-645c7b811ff3 service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:30.237350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8706537-61d7-4b34-b99f-645c7b811ff3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:30.239112 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-d8706537-61d7-4b34-b99f-645c7b811ff3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:30.261764 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:30:30.386652 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8706537-61d7-4b34-b99f-645c7b811ff3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:30:30.388250 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8706537-61d7-4b34-b99f-645c7b811ff3 service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:30:30.410098 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 665/1326] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:30:29 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 413 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:30:32.612428 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70b3a90-c95e-4d7e-99a8-96fb1246e43e None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:32.612969 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70b3a90-c95e-4d7e-99a8-96fb1246e43e None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:32.612969 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70b3a90-c95e-4d7e-99a8-96fb1246e43e None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:32.614358 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 662/1327] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:30:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:30:32.620852 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1851657-a718-4b08-b10b-61e36ab2e479 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:32.621344 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1851657-a718-4b08-b10b-61e36ab2e479 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:32.621529 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1851657-a718-4b08-b10b-61e36ab2e479 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:32.701525 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-b1851657-a718-4b08-b10b-61e36ab2e479 None None] MFA Rules not processed for user `dbc90e3cb55f431e91cd23da7b8958f2`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:30:32.702884 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-b1851657-a718-4b08-b10b-61e36ab2e479 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:32.785949 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:30:32.826405 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 666/1328] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:30:32 2022] POST /identity/v3/auth/tokens => generated 3233 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:30:32.867451 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70b3a90-c95e-4d7e-99a8-96fb1246e43e None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:30:32.869053 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-c70b3a90-c95e-4d7e-99a8-96fb1246e43e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:32.906294 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:30:33.000475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70b3a90-c95e-4d7e-99a8-96fb1246e43e None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:30:33.102361 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bf9de1-ac5b-4b0b-a2b2-d8e28bb76ee6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:30:33.103134 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bf9de1-ac5b-4b0b-a2b2-d8e28bb76ee6 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:33.103418 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bf9de1-ac5b-4b0b-a2b2-d8e28bb76ee6 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:33.103976 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bf9de1-ac5b-4b0b-a2b2-d8e28bb76ee6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:33.105853 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-e8bf9de1-ac5b-4b0b-a2b2-d8e28bb76ee6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:33.198060 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:30:33.238402 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8bf9de1-ac5b-4b0b-a2b2-d8e28bb76ee6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:30:33.249794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8bf9de1-ac5b-4b0b-a2b2-d8e28bb76ee6 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:30:33.273712 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 663/1329] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:30:32 2022] GET /identity/v3/auth/tokens => generated 3233 bytes in 407 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:30:35.994600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10cae40-ee1a-4620-86dc-58f3527975d4 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:35.995143 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10cae40-ee1a-4620-86dc-58f3527975d4 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:35.995143 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10cae40-ee1a-4620-86dc-58f3527975d4 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:35.996552 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 667/1330] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:30:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:30:36.002006 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a652681-d64d-4155-87fa-864c861d3fae None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:36.002423 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a652681-d64d-4155-87fa-864c861d3fae None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:36.002693 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a652681-d64d-4155-87fa-864c861d3fae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:36.071472 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-2a652681-d64d-4155-87fa-864c861d3fae None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:30:36.072708 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-2a652681-d64d-4155-87fa-864c861d3fae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:36.173805 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 664/1331] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:30:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:30:36.210828 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10cae40-ee1a-4620-86dc-58f3527975d4 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:30:36.212376 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-c10cae40-ee1a-4620-86dc-58f3527975d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:36.318368 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c10cae40-ee1a-4620-86dc-58f3527975d4 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:30:36.395675 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-366e9dd9-2a0f-4540-af81-60955e7016cc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:30:36.396756 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366e9dd9-2a0f-4540-af81-60955e7016cc service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:36.397040 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366e9dd9-2a0f-4540-af81-60955e7016cc service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:36.397313 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-366e9dd9-2a0f-4540-af81-60955e7016cc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:36.399081 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-366e9dd9-2a0f-4540-af81-60955e7016cc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:36.426507 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:30:36.510666 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366e9dd9-2a0f-4540-af81-60955e7016cc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:30:36.512110 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-366e9dd9-2a0f-4540-af81-60955e7016cc service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:30:36.530655 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 668/1332] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:30:36 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:30:38.643217 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d14003-5799-4452-97d0-73876e59b3a9 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:38.643750 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d14003-5799-4452-97d0-73876e59b3a9 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:38.643946 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1d14003-5799-4452-97d0-73876e59b3a9 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:38.645248 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 665/1333] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:30:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:30:38.651707 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3246324-5dfe-432d-915c-add1aab46487 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:38.652251 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3246324-5dfe-432d-915c-add1aab46487 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:38.652350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3246324-5dfe-432d-915c-add1aab46487 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:38.718580 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-f3246324-5dfe-432d-915c-add1aab46487 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:30:38.719838 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-f3246324-5dfe-432d-915c-add1aab46487 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:38.797441 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:30:38.820006 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 669/1334] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:30:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:30:38.860624 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d14003-5799-4452-97d0-73876e59b3a9 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:30:38.862162 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-c1d14003-5799-4452-97d0-73876e59b3a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:38.903036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:30:38.969986 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1d14003-5799-4452-97d0-73876e59b3a9 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:30:39.045330 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36656055-ecfc-474e-a1cd-497c59e8d8b5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:30:39.045989 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36656055-ecfc-474e-a1cd-497c59e8d8b5 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:39.046278 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36656055-ecfc-474e-a1cd-497c59e8d8b5 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:39.046592 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36656055-ecfc-474e-a1cd-497c59e8d8b5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:39.048271 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-36656055-ecfc-474e-a1cd-497c59e8d8b5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:39.136531 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:30:39.164957 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36656055-ecfc-474e-a1cd-497c59e8d8b5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:30:39.166400 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36656055-ecfc-474e-a1cd-497c59e8d8b5 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:30:39.186716 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 666/1335] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:30:38 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:30:42.030411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40417239-3d95-4dd4-9fce-085da18cdc1d None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:42.031029 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40417239-3d95-4dd4-9fce-085da18cdc1d None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:42.031029 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40417239-3d95-4dd4-9fce-085da18cdc1d None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:42.032415 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 670/1336] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:30:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:30:42.038509 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29cfe1ca-2437-4227-a0a0-e8c847fbe0ad None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:42.039059 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29cfe1ca-2437-4227-a0a0-e8c847fbe0ad None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:42.039059 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29cfe1ca-2437-4227-a0a0-e8c847fbe0ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:42.113250 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-29cfe1ca-2437-4227-a0a0-e8c847fbe0ad None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:30:42.115171 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-29cfe1ca-2437-4227-a0a0-e8c847fbe0ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:42.222761 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 667/1337] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:30:42 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:30:42.260408 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40417239-3d95-4dd4-9fce-085da18cdc1d None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:30:42.261991 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-40417239-3d95-4dd4-9fce-085da18cdc1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:42.377467 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40417239-3d95-4dd4-9fce-085da18cdc1d None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:30:42.463483 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be883a70-83f3-4b8a-b544-9b7ff54e0de0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:30:42.464136 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be883a70-83f3-4b8a-b544-9b7ff54e0de0 service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:42.464459 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be883a70-83f3-4b8a-b544-9b7ff54e0de0 service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:42.464730 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be883a70-83f3-4b8a-b544-9b7ff54e0de0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:42.466474 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-be883a70-83f3-4b8a-b544-9b7ff54e0de0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:42.509009 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:30:42.592609 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be883a70-83f3-4b8a-b544-9b7ff54e0de0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:30:42.594178 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be883a70-83f3-4b8a-b544-9b7ff54e0de0 service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:30:42.615606 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 671/1338] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:30:42 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:30:47.234771 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06581abd-6e71-4015-b711-cd070f576029 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:47.235469 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06581abd-6e71-4015-b711-cd070f576029 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:47.235713 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06581abd-6e71-4015-b711-cd070f576029 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:47.237562 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 668/1339] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:30:47 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 12 17:30:47.243785 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a46655-c7cd-466c-b331-c409c6ac3dba None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:47.244367 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a46655-c7cd-466c-b331-c409c6ac3dba None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:47.244367 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a46655-c7cd-466c-b331-c409c6ac3dba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:47.323277 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-36a46655-c7cd-466c-b331-c409c6ac3dba None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:30:47.324513 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-36a46655-c7cd-466c-b331-c409c6ac3dba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:47.436389 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:30:47.450522 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 672/1340] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:30:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:30:47.508088 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109aa0ec-31dd-4c1b-baef-2eda996b6d3c None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:47.508474 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109aa0ec-31dd-4c1b-baef-2eda996b6d3c None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:47.508870 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109aa0ec-31dd-4c1b-baef-2eda996b6d3c None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:47.511371 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 669/1341] 10.222.0.16 () {54 vars in 964 bytes} [Tue Jul 12 17:30:47 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:30:47.520294 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3861419-fcc2-42e7-9e4f-fdf43fa8d9aa None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:47.521221 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3861419-fcc2-42e7-9e4f-fdf43fa8d9aa None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:47.521564 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3861419-fcc2-42e7-9e4f-fdf43fa8d9aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:47.618398 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-e3861419-fcc2-42e7-9e4f-fdf43fa8d9aa None None] MFA Rules not processed for user `bf5bf02ae13d4d86a1b157369ef07a00`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:30:47.619646 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-e3861419-fcc2-42e7-9e4f-fdf43fa8d9aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:47.702659 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 673/1342] 10.222.0.16 () {58 vars in 1061 bytes} [Tue Jul 12 17:30:47 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:30:47.711632 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109aa0ec-31dd-4c1b-baef-2eda996b6d3c None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:30:47.713165 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-109aa0ec-31dd-4c1b-baef-2eda996b6d3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:47.766743 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:30:47.822762 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109aa0ec-31dd-4c1b-baef-2eda996b6d3c None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:30:47.900344 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ca8649-8e58-4e74-a5ba-922add78df6b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:30:47.900995 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ca8649-8e58-4e74-a5ba-922add78df6b service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:47.901289 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ca8649-8e58-4e74-a5ba-922add78df6b service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:47.901566 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ca8649-8e58-4e74-a5ba-922add78df6b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:47.903304 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-53ca8649-8e58-4e74-a5ba-922add78df6b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:48.006367 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:30:48.023597 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53ca8649-8e58-4e74-a5ba-922add78df6b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:30:48.024873 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53ca8649-8e58-4e74-a5ba-922add78df6b service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:30:48.044024 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 670/1343] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:30:47 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:30:51.700322 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e38f58-5b8d-4dfe-bebb-092aa779b6b3 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:51.700892 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e38f58-5b8d-4dfe-bebb-092aa779b6b3 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:51.700892 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e38f58-5b8d-4dfe-bebb-092aa779b6b3 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:51.702292 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 674/1344] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:30:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:30:51.709366 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4c4baf-c229-4617-a1fa-15ef99f64915 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:51.709859 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4c4baf-c229-4617-a1fa-15ef99f64915 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:51.709963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4c4baf-c229-4617-a1fa-15ef99f64915 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:51.804485 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-ce4c4baf-c229-4617-a1fa-15ef99f64915 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:30:51.805728 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-ce4c4baf-c229-4617-a1fa-15ef99f64915 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:51.916611 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 671/1345] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:30:51 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:30:51.955745 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e38f58-5b8d-4dfe-bebb-092aa779b6b3 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:30:51.957370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-49e38f58-5b8d-4dfe-bebb-092aa779b6b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:52.049892 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:30:52.069879 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e38f58-5b8d-4dfe-bebb-092aa779b6b3 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:30:52.148167 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db46982c-8632-47c5-b365-5fb6e66c18ae service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:30:52.148817 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db46982c-8632-47c5-b365-5fb6e66c18ae service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:52.149099 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db46982c-8632-47c5-b365-5fb6e66c18ae service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:52.149375 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db46982c-8632-47c5-b365-5fb6e66c18ae service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:52.151070 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-db46982c-8632-47c5-b365-5fb6e66c18ae service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:52.283174 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db46982c-8632-47c5-b365-5fb6e66c18ae service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:30:52.284696 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db46982c-8632-47c5-b365-5fb6e66c18ae service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:30:52.309759 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 675/1346] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:30:51 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 354 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:30:57.025541 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca600612-1385-4796-ad4e-14018ec3dbc1 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:57.026088 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca600612-1385-4796-ad4e-14018ec3dbc1 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:57.026088 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca600612-1385-4796-ad4e-14018ec3dbc1 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:57.027684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 672/1347] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:30:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:30:57.032841 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8460ddff-c019-48c6-ab98-cf2810015588 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:57.033276 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8460ddff-c019-48c6-ab98-cf2810015588 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:57.033413 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8460ddff-c019-48c6-ab98-cf2810015588 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:57.049483 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:30:57.116723 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-8460ddff-c019-48c6-ab98-cf2810015588 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:30:57.117980 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-8460ddff-c019-48c6-ab98-cf2810015588 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:57.241260 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 676/1348] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:30:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:30:57.281324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca600612-1385-4796-ad4e-14018ec3dbc1 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:30:57.282939 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-ca600612-1385-4796-ad4e-14018ec3dbc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:57.351283 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:30:57.400828 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca600612-1385-4796-ad4e-14018ec3dbc1 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:30:57.476945 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-499d7f7e-10bd-4e46-870f-747c6321044c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:30:57.477600 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499d7f7e-10bd-4e46-870f-747c6321044c service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:30:57.477888 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499d7f7e-10bd-4e46-870f-747c6321044c service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:30:57.478172 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-499d7f7e-10bd-4e46-870f-747c6321044c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:30:57.479941 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-499d7f7e-10bd-4e46-870f-747c6321044c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:30:57.589152 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:30:57.592798 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-499d7f7e-10bd-4e46-870f-747c6321044c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:30:57.594180 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-499d7f7e-10bd-4e46-870f-747c6321044c service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:30:57.619464 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 673/1349] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:30:57 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:31:02.760155 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93857ceb-4992-45ed-8d69-9eda8748a2ba None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:02.760647 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93857ceb-4992-45ed-8d69-9eda8748a2ba None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:02.760754 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93857ceb-4992-45ed-8d69-9eda8748a2ba None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:02.764973 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 677/1350] 10.222.0.16 () {54 vars in 921 bytes} [Tue Jul 12 17:31:02 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:31:02.772040 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c93e2c6-df63-4995-b5b3-e0930cdd5b01 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:02.772522 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c93e2c6-df63-4995-b5b3-e0930cdd5b01 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:02.772605 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c93e2c6-df63-4995-b5b3-e0930cdd5b01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:02.860711 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-8c93e2c6-df63-4995-b5b3-e0930cdd5b01 None None] MFA Rules not processed for user `f623be28c83247c78d3ea6873c347fe0`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:31:02.861907 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-8c93e2c6-df63-4995-b5b3-e0930cdd5b01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:02.981993 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 674/1351] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Jul 12 17:31:02 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:31:07.216102 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7a3579-feee-4c72-9991-3104d0a47dde None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:07.216675 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7a3579-feee-4c72-9991-3104d0a47dde None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:07.216675 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7a3579-feee-4c72-9991-3104d0a47dde None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:07.218045 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 678/1352] 10.222.0.16 () {54 vars in 921 bytes} [Tue Jul 12 17:31:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:31:07.223755 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f127f523-c22e-4c19-a19b-18d7109b7c4b None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:07.224569 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f127f523-c22e-4c19-a19b-18d7109b7c4b None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:07.225032 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f127f523-c22e-4c19-a19b-18d7109b7c4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:07.238780 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=59 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:07.289472 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:07.314466 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-f127f523-c22e-4c19-a19b-18d7109b7c4b None None] MFA Rules not processed for user `f623be28c83247c78d3ea6873c347fe0`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:31:07.316296 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-f127f523-c22e-4c19-a19b-18d7109b7c4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:07.438036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 675/1353] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Jul 12 17:31:07 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:31:08.934572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6188d1-5f55-4c03-8121-f6195a3bd8c3 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:08.935073 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6188d1-5f55-4c03-8121-f6195a3bd8c3 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:08.935073 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6188d1-5f55-4c03-8121-f6195a3bd8c3 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:08.936485 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 679/1354] 10.222.0.16 () {54 vars in 921 bytes} [Tue Jul 12 17:31:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:31:08.942607 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9fa3d7-f8c8-4c38-8722-5826c8fc5cf4 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:08.943085 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9fa3d7-f8c8-4c38-8722-5826c8fc5cf4 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:08.943313 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d9fa3d7-f8c8-4c38-8722-5826c8fc5cf4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:09.022369 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-2d9fa3d7-f8c8-4c38-8722-5826c8fc5cf4 None None] MFA Rules not processed for user `f623be28c83247c78d3ea6873c347fe0`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:31:09.023783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-2d9fa3d7-f8c8-4c38-8722-5826c8fc5cf4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:09.068533 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:09.136150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 676/1355] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Jul 12 17:31:08 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:31:12.567916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2c5018-d48f-49c7-9468-59663346a0ff None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:12.570041 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2c5018-d48f-49c7-9468-59663346a0ff None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:12.570562 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2c5018-d48f-49c7-9468-59663346a0ff None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:12.580815 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 680/1356] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:31:12 2022] GET /identity => generated 270 bytes in 18 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:31:12.587290 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e468a7bf-47aa-4a80-950e-0bd1da3a68b8 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:12.588173 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e468a7bf-47aa-4a80-950e-0bd1da3a68b8 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:12.589234 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e468a7bf-47aa-4a80-950e-0bd1da3a68b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:12.682061 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-e468a7bf-47aa-4a80-950e-0bd1da3a68b8 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:31:12.683764 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-e468a7bf-47aa-4a80-950e-0bd1da3a68b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:12.806669 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 677/1357] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:31:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:31:12.864628 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d418858-3456-4bbd-b541-d948ce9ee130 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:12.865648 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d418858-3456-4bbd-b541-d948ce9ee130 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:12.866163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d418858-3456-4bbd-b541-d948ce9ee130 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:12.868568 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 681/1358] 10.222.0.16 () {54 vars in 961 bytes} [Tue Jul 12 17:31:12 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:31:12.877177 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3954c95a-e6ad-45a3-bb60-a9e40066b7ca None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:12.877881 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3954c95a-e6ad-45a3-bb60-a9e40066b7ca None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:12.878805 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3954c95a-e6ad-45a3-bb60-a9e40066b7ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:12.900414 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:12.985941 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-3954c95a-e6ad-45a3-bb60-a9e40066b7ca None None] MFA Rules not processed for user `9012d5b88f554dc596582a7693c2a72a`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:31:12.987460 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-3954c95a-e6ad-45a3-bb60-a9e40066b7ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:13.123687 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 678/1359] 10.222.0.16 () {58 vars in 1058 bytes} [Tue Jul 12 17:31:12 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 248 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:31:13.131720 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d418858-3456-4bbd-b541-d948ce9ee130 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:31:13.133352 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-2d418858-3456-4bbd-b541-d948ce9ee130 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:13.306611 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d418858-3456-4bbd-b541-d948ce9ee130 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:31:13.386604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:13.432082 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e36b070-5945-4bf4-b05b-196daa846d42 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:31:13.432742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e36b070-5945-4bf4-b05b-196daa846d42 service nova] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:13.433028 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e36b070-5945-4bf4-b05b-196daa846d42 service nova] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:13.433307 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e36b070-5945-4bf4-b05b-196daa846d42 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:13.435248 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-3e36b070-5945-4bf4-b05b-196daa846d42 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:13.540327 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e36b070-5945-4bf4-b05b-196daa846d42 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:31:13.541970 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e36b070-5945-4bf4-b05b-196daa846d42 service nova] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:31:13.562257 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 682/1360] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:31:13 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 431 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:31:15.909532 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe36430-0f85-49ea-8206-347de852ddd3 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:15.910348 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe36430-0f85-49ea-8206-347de852ddd3 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:15.910348 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe36430-0f85-49ea-8206-347de852ddd3 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:15.912316 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 679/1361] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:31:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:31:15.918303 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593fc481-8ab0-489f-8e29-c06afe8decbd None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:15.919247 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593fc481-8ab0-489f-8e29-c06afe8decbd None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:15.919536 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593fc481-8ab0-489f-8e29-c06afe8decbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:15.993566 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-593fc481-8ab0-489f-8e29-c06afe8decbd None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:31:15.994860 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-593fc481-8ab0-489f-8e29-c06afe8decbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:16.035956 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:16.100562 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 683/1362] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:31:15 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:31:16.145411 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe36430-0f85-49ea-8206-347de852ddd3 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:31:16.146954 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-bbe36430-0f85-49ea-8206-347de852ddd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:16.231344 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:16.342412 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe36430-0f85-49ea-8206-347de852ddd3 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:31:16.480562 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91add91-8195-4b62-abb7-7dcd80df425d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:31:16.487247 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91add91-8195-4b62-abb7-7dcd80df425d service nova] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:16.487247 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91add91-8195-4b62-abb7-7dcd80df425d service nova] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:16.487247 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91add91-8195-4b62-abb7-7dcd80df425d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:16.487247 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-b91add91-8195-4b62-abb7-7dcd80df425d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:16.511233 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:16.593458 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91add91-8195-4b62-abb7-7dcd80df425d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:31:16.595369 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91add91-8195-4b62-abb7-7dcd80df425d service nova] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:31:16.616134 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 680/1363] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:31:16 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 471 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:31:19.282640 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57e3f17-c3a2-46a1-ab37-37bd2851ce85 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:19.283224 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57e3f17-c3a2-46a1-ab37-37bd2851ce85 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:19.283224 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57e3f17-c3a2-46a1-ab37-37bd2851ce85 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:19.284629 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 684/1364] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:31:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:31:19.290066 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e151e1-8703-453d-b072-4542e7818c1f None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:19.290885 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e151e1-8703-453d-b072-4542e7818c1f None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:19.291171 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e151e1-8703-453d-b072-4542e7818c1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:19.363520 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-e8e151e1-8703-453d-b072-4542e7818c1f None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:31:19.365045 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-e8e151e1-8703-453d-b072-4542e7818c1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:19.469474 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:19.478485 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 681/1365] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:31:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:31:19.522029 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3d262c-6604-4db6-b9c0-a735389c54a5 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:19.522704 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3d262c-6604-4db6-b9c0-a735389c54a5 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:19.523087 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3d262c-6604-4db6-b9c0-a735389c54a5 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:19.524799 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 685/1366] 10.222.0.16 () {54 vars in 961 bytes} [Tue Jul 12 17:31:19 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:31:19.530268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d807746-de2a-41d6-a63c-0fcb81d8032d None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:19.530588 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d807746-de2a-41d6-a63c-0fcb81d8032d None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:19.530855 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d807746-de2a-41d6-a63c-0fcb81d8032d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:19.617120 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-3d807746-de2a-41d6-a63c-0fcb81d8032d None None] MFA Rules not processed for user `9012d5b88f554dc596582a7693c2a72a`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:31:19.618281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-3d807746-de2a-41d6-a63c-0fcb81d8032d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:19.755811 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 682/1367] 10.222.0.16 () {58 vars in 1058 bytes} [Tue Jul 12 17:31:19 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:31:19.763926 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3d262c-6604-4db6-b9c0-a735389c54a5 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:31:19.765822 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-2b3d262c-6604-4db6-b9c0-a735389c54a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:19.933284 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3d262c-6604-4db6-b9c0-a735389c54a5 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:31:19.956897 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:20.070415 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7382ab17-543d-4dd8-909a-1d0e788d5bf7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:31:20.071107 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7382ab17-543d-4dd8-909a-1d0e788d5bf7 service nova] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:20.071397 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7382ab17-543d-4dd8-909a-1d0e788d5bf7 service nova] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:20.071699 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7382ab17-543d-4dd8-909a-1d0e788d5bf7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:20.073380 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-7382ab17-543d-4dd8-909a-1d0e788d5bf7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:20.180733 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7382ab17-543d-4dd8-909a-1d0e788d5bf7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:31:20.183065 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7382ab17-543d-4dd8-909a-1d0e788d5bf7 service nova] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:31:20.204076 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 686/1368] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:31:19 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 440 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:31:21.085721 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df1f20f-f2fa-4b7b-b11e-7bd47edc1407 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:21.086296 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df1f20f-f2fa-4b7b-b11e-7bd47edc1407 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:21.086296 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df1f20f-f2fa-4b7b-b11e-7bd47edc1407 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:21.087743 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 683/1369] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:31:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:31:21.092847 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e37ab3-3b6b-47d2-93fa-cdd6eb8cfe12 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:21.093275 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e37ab3-3b6b-47d2-93fa-cdd6eb8cfe12 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:21.093476 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e37ab3-3b6b-47d2-93fa-cdd6eb8cfe12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:21.126769 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:21.164045 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-05e37ab3-3b6b-47d2-93fa-cdd6eb8cfe12 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:31:21.165283 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-05e37ab3-3b6b-47d2-93fa-cdd6eb8cfe12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:21.271169 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 687/1370] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:31:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:31:21.626171 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ae106d-5476-4014-b979-44491672d43f None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:21.626171 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ae106d-5476-4014-b979-44491672d43f None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:21.626171 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ae106d-5476-4014-b979-44491672d43f None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:21.626171 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 684/1371] 10.222.0.16 () {54 vars in 963 bytes} [Tue Jul 12 17:31:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:31:21.631511 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e18d0d1-8524-4e6b-9d34-501c0c07f14f None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:21.631886 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e18d0d1-8524-4e6b-9d34-501c0c07f14f None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:21.632350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e18d0d1-8524-4e6b-9d34-501c0c07f14f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:21.717213 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-7e18d0d1-8524-4e6b-9d34-501c0c07f14f None None] MFA Rules not processed for user `f6a82b49464846d1a207189f17fd4cf7`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:31:21.718508 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-7e18d0d1-8524-4e6b-9d34-501c0c07f14f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:21.769302 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:21.806356 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 688/1372] 10.222.0.16 () {58 vars in 1060 bytes} [Tue Jul 12 17:31:21 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:31:21.807667 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ae106d-5476-4014-b979-44491672d43f None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:31:21.809290 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-d2ae106d-5476-4014-b979-44491672d43f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:21.867181 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:21.928025 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ae106d-5476-4014-b979-44491672d43f None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:31:22.011243 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de04af0b-f521-4c52-a153-543dea49bf75 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:31:22.011898 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de04af0b-f521-4c52-a153-543dea49bf75 service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:22.012190 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de04af0b-f521-4c52-a153-543dea49bf75 service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:22.012463 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de04af0b-f521-4c52-a153-543dea49bf75 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:22.014310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-de04af0b-f521-4c52-a153-543dea49bf75 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:22.129000 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:22.147393 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de04af0b-f521-4c52-a153-543dea49bf75 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:31:22.149163 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de04af0b-f521-4c52-a153-543dea49bf75 service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:31:22.168801 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 685/1373] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:31:21 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:31:22.498824 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d5dec8-d414-40d0-9ded-b3c79c3f7ac1 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:22.498824 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d5dec8-d414-40d0-9ded-b3c79c3f7ac1 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:22.498824 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d5dec8-d414-40d0-9ded-b3c79c3f7ac1 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:22.504320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 689/1374] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:31:22 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:31:22.505754 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429c4051-3458-4523-81ce-70669422b8fc None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:22.505945 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429c4051-3458-4523-81ce-70669422b8fc None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:22.506491 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429c4051-3458-4523-81ce-70669422b8fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:22.575959 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-429c4051-3458-4523-81ce-70669422b8fc None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:31:22.576775 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-429c4051-3458-4523-81ce-70669422b8fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:22.685484 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 686/1375] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:31:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:31:22.729585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d5dec8-d414-40d0-9ded-b3c79c3f7ac1 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:31:22.731134 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-82d5dec8-d414-40d0-9ded-b3c79c3f7ac1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:22.881112 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d5dec8-d414-40d0-9ded-b3c79c3f7ac1 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:31:22.937130 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:23.030658 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09805cf6-aa69-41dd-9d32-911596286183 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:31:23.031309 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09805cf6-aa69-41dd-9d32-911596286183 service nova] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:23.031596 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09805cf6-aa69-41dd-9d32-911596286183 service nova] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:23.031876 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09805cf6-aa69-41dd-9d32-911596286183 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:23.033568 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-09805cf6-aa69-41dd-9d32-911596286183 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:23.129488 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09805cf6-aa69-41dd-9d32-911596286183 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:31:23.131128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09805cf6-aa69-41dd-9d32-911596286183 service nova] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:31:23.149534 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 690/1376] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:31:22 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 420 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:31:25.364854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba8c5ab-e379-46fe-8a22-2977bae3f300 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:25.365355 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba8c5ab-e379-46fe-8a22-2977bae3f300 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:25.365454 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba8c5ab-e379-46fe-8a22-2977bae3f300 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:25.367097 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 687/1377] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:31:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:31:25.372487 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d1c4fc-ea01-48bb-86fe-d436f1db5d0f None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:25.372912 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d1c4fc-ea01-48bb-86fe-d436f1db5d0f None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:25.373118 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d1c4fc-ea01-48bb-86fe-d436f1db5d0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:25.454081 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-72d1c4fc-ea01-48bb-86fe-d436f1db5d0f None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:31:25.455380 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-72d1c4fc-ea01-48bb-86fe-d436f1db5d0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:25.463170 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:25.558616 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 691/1378] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:31:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:31:25.604616 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba8c5ab-e379-46fe-8a22-2977bae3f300 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:31:25.606150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-cba8c5ab-e379-46fe-8a22-2977bae3f300 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:25.681853 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:25.787222 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba8c5ab-e379-46fe-8a22-2977bae3f300 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:31:25.949216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf697ad-ce91-4f4d-95fd-da6c81a387b4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:31:25.950090 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf697ad-ce91-4f4d-95fd-da6c81a387b4 service nova] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:25.950392 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf697ad-ce91-4f4d-95fd-da6c81a387b4 service nova] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:25.950720 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf697ad-ce91-4f4d-95fd-da6c81a387b4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:25.952579 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-aaf697ad-ce91-4f4d-95fd-da6c81a387b4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:25.975924 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:26.060650 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf697ad-ce91-4f4d-95fd-da6c81a387b4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:31:26.062633 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf697ad-ce91-4f4d-95fd-da6c81a387b4 service nova] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:31:26.083352 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 688/1379] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:31:25 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 479 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:31:28.287647 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd5767c-6fb8-43a7-abc6-3121ee27a427 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:28.287647 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd5767c-6fb8-43a7-abc6-3121ee27a427 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:28.287647 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd5767c-6fb8-43a7-abc6-3121ee27a427 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:28.295052 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bada84f-404e-4bd4-9d36-e8db48fcc8d6 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:28.295052 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bada84f-404e-4bd4-9d36-e8db48fcc8d6 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:28.295052 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bada84f-404e-4bd4-9d36-e8db48fcc8d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:28.298717 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 692/1380] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:31:28 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 12 17:31:28.364547 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-0bada84f-404e-4bd4-9d36-e8db48fcc8d6 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:31:28.375348 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-0bada84f-404e-4bd4-9d36-e8db48fcc8d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:28.461121 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:28.467411 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 689/1381] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:31:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:31:28.512768 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd5767c-6fb8-43a7-abc6-3121ee27a427 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:31:28.514513 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-ddd5767c-6fb8-43a7-abc6-3121ee27a427 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:28.670865 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:28.683661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd5767c-6fb8-43a7-abc6-3121ee27a427 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:31:28.813557 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a820b1ae-0725-4eed-810f-00c04e197b04 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:31:28.814118 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a820b1ae-0725-4eed-810f-00c04e197b04 service nova] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:28.814404 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a820b1ae-0725-4eed-810f-00c04e197b04 service nova] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:28.814717 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a820b1ae-0725-4eed-810f-00c04e197b04 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:28.817519 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-a820b1ae-0725-4eed-810f-00c04e197b04 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:28.921297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a820b1ae-0725-4eed-810f-00c04e197b04 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:31:28.923173 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a820b1ae-0725-4eed-810f-00c04e197b04 service nova] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:31:28.937285 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:28.948322 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 693/1382] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:31:28 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 436 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:31:31.278196 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60029ffe-cb49-4058-af2b-e625c7c02aa2 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:31.278762 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60029ffe-cb49-4058-af2b-e625c7c02aa2 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:31.278999 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60029ffe-cb49-4058-af2b-e625c7c02aa2 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:31.280756 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 690/1383] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:31:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:31:31.286679 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b5e447-a086-481e-b1fb-cb620f7a56b1 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:31.287131 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b5e447-a086-481e-b1fb-cb620f7a56b1 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:31.287402 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b5e447-a086-481e-b1fb-cb620f7a56b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:31.365378 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-78b5e447-a086-481e-b1fb-cb620f7a56b1 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:31:31.366578 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-78b5e447-a086-481e-b1fb-cb620f7a56b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:31.481931 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 694/1384] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:31:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:31:31.526935 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60029ffe-cb49-4058-af2b-e625c7c02aa2 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:31:31.528914 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-60029ffe-cb49-4058-af2b-e625c7c02aa2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:31.710649 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60029ffe-cb49-4058-af2b-e625c7c02aa2 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:31:31.796120 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:31.844462 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0520f2-2c84-41e8-977a-5496a66768ec service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:31:31.845375 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0520f2-2c84-41e8-977a-5496a66768ec service nova] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:31.845777 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0520f2-2c84-41e8-977a-5496a66768ec service nova] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:31.846244 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0520f2-2c84-41e8-977a-5496a66768ec service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:31.848453 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-7b0520f2-2c84-41e8-977a-5496a66768ec service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:31.940995 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b0520f2-2c84-41e8-977a-5496a66768ec service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:31:31.942629 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b0520f2-2c84-41e8-977a-5496a66768ec service nova] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:31:31.961044 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 691/1385] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:31:31 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 435 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:31:34.029566 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559c5b71-4703-4ce2-b6c6-59446d966536 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:34.030322 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559c5b71-4703-4ce2-b6c6-59446d966536 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:34.030322 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559c5b71-4703-4ce2-b6c6-59446d966536 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:34.031556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 695/1386] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:31:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:31:34.037465 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6ac656-76e5-4203-a169-7ffd58708215 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:34.037909 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6ac656-76e5-4203-a169-7ffd58708215 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:34.038031 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6ac656-76e5-4203-a169-7ffd58708215 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:34.105491 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-3d6ac656-76e5-4203-a169-7ffd58708215 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:31:34.106697 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-3d6ac656-76e5-4203-a169-7ffd58708215 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:34.142248 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:34.229773 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 692/1387] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:31:34 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:31:34.276616 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559c5b71-4703-4ce2-b6c6-59446d966536 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:31:34.278131 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-559c5b71-4703-4ce2-b6c6-59446d966536 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:34.363069 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:34.441682 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559c5b71-4703-4ce2-b6c6-59446d966536 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:31:34.566557 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3ebe4e-e6b0-4e06-bd28-4f885045786e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:31:34.566557 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3ebe4e-e6b0-4e06-bd28-4f885045786e service nova] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:34.566557 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3ebe4e-e6b0-4e06-bd28-4f885045786e service nova] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:34.566557 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3ebe4e-e6b0-4e06-bd28-4f885045786e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:34.568881 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-6c3ebe4e-e6b0-4e06-bd28-4f885045786e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:34.616040 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:34.671320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c3ebe4e-e6b0-4e06-bd28-4f885045786e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:31:34.672953 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c3ebe4e-e6b0-4e06-bd28-4f885045786e service nova] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:31:34.692952 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 696/1388] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:31:34 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 417 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:31:37.139416 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2886f501-e4e2-4589-a290-2a14ccf777b0 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:37.139999 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2886f501-e4e2-4589-a290-2a14ccf777b0 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:37.139999 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2886f501-e4e2-4589-a290-2a14ccf777b0 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:37.141363 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 693/1389] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:31:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:31:37.146873 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc850d2-890f-4f3e-8bbe-716959dc98ab None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:37.147289 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc850d2-890f-4f3e-8bbe-716959dc98ab None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:37.147439 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc850d2-890f-4f3e-8bbe-716959dc98ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:37.215497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-adc850d2-890f-4f3e-8bbe-716959dc98ab None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:31:37.216714 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-adc850d2-890f-4f3e-8bbe-716959dc98ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:37.315841 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 697/1390] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:31:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:31:37.359106 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2886f501-e4e2-4589-a290-2a14ccf777b0 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:31:37.360660 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-2886f501-e4e2-4589-a290-2a14ccf777b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:37.510800 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2886f501-e4e2-4589-a290-2a14ccf777b0 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:31:37.527973 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:37.639009 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dac6c38-5a3a-4f34-bc04-e3e24533e75c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:31:37.639727 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dac6c38-5a3a-4f34-bc04-e3e24533e75c service nova] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:37.640029 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dac6c38-5a3a-4f34-bc04-e3e24533e75c service nova] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:37.640306 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dac6c38-5a3a-4f34-bc04-e3e24533e75c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:37.642212 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-5dac6c38-5a3a-4f34-bc04-e3e24533e75c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:37.735271 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dac6c38-5a3a-4f34-bc04-e3e24533e75c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:31:37.737454 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dac6c38-5a3a-4f34-bc04-e3e24533e75c service nova] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:31:37.757918 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 694/1391] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:31:37 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 399 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:31:38.025663 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e67515c-8a7d-425f-9793-dc7ca831739f None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:38.025952 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e67515c-8a7d-425f-9793-dc7ca831739f None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:38.026236 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e67515c-8a7d-425f-9793-dc7ca831739f None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:38.027681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 698/1392] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:31:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:31:38.033891 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1780e7e-71ec-4ac7-82f9-ed8c752a8b1e None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:38.034185 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1780e7e-71ec-4ac7-82f9-ed8c752a8b1e None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:38.034494 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1780e7e-71ec-4ac7-82f9-ed8c752a8b1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:38.062499 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:38.107547 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-f1780e7e-71ec-4ac7-82f9-ed8c752a8b1e None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:31:38.108692 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-f1780e7e-71ec-4ac7-82f9-ed8c752a8b1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:38.209032 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 695/1393] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:31:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:31:38.264922 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e33952-1500-4649-978d-98533f9f878d None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:38.265451 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e33952-1500-4649-978d-98533f9f878d None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:38.265534 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e33952-1500-4649-978d-98533f9f878d None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:38.266900 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 699/1394] 10.222.0.16 () {54 vars in 962 bytes} [Tue Jul 12 17:31:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:31:38.272442 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890215f8-753c-42cc-bc55-89214c0c262b None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:38.272736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890215f8-753c-42cc-bc55-89214c0c262b None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:38.273018 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890215f8-753c-42cc-bc55-89214c0c262b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:38.350236 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-890215f8-753c-42cc-bc55-89214c0c262b None None] MFA Rules not processed for user `4018d94a1feb4d6dade88ee9663f9729`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:31:38.351503 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-890215f8-753c-42cc-bc55-89214c0c262b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:38.394201 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:38.432326 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 696/1395] 10.222.0.16 () {58 vars in 1059 bytes} [Tue Jul 12 17:31:38 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:31:38.440424 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e33952-1500-4649-978d-98533f9f878d None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:31:38.442008 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-97e33952-1500-4649-978d-98533f9f878d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:38.455975 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:38.550339 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e33952-1500-4649-978d-98533f9f878d None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:31:38.643895 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc2c98af-772c-474b-a2d3-649df4e73c2d service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4018d94a1feb4d6dade88ee9663f9729', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:31:38.644820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2c98af-772c-474b-a2d3-649df4e73c2d service glance] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:38.645043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2c98af-772c-474b-a2d3-649df4e73c2d service glance] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:38.645321 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2c98af-772c-474b-a2d3-649df4e73c2d service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:38.647062 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-fc2c98af-772c-474b-a2d3-649df4e73c2d service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:38.724118 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:38.768529 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc2c98af-772c-474b-a2d3-649df4e73c2d service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:31:38.770352 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc2c98af-772c-474b-a2d3-649df4e73c2d service glance] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:31:38.791354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 700/1396] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:31:38 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:31:39.697517 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d02dc5d-9330-407b-8d51-dab23ac596da None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:39.698006 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d02dc5d-9330-407b-8d51-dab23ac596da None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:39.698099 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d02dc5d-9330-407b-8d51-dab23ac596da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:39.769997 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-9d02dc5d-9330-407b-8d51-dab23ac596da None None] MFA Rules not processed for user `3ac0d5d8de4943a3888abee3b24b8a6f`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:31:39.771296 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-9d02dc5d-9330-407b-8d51-dab23ac596da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:39.861365 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 697/1397] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Jul 12 17:31:39 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:31:39.910555 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3864e77e-6f13-4522-ab8a-290ec238fcaa None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:39.911074 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3864e77e-6f13-4522-ab8a-290ec238fcaa None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:39.911179 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3864e77e-6f13-4522-ab8a-290ec238fcaa None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:39.912552 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 701/1398] 10.222.0.16 () {54 vars in 942 bytes} [Tue Jul 12 17:31:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:31:39.919195 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2596e56-bf15-4cb8-92e2-5bc00d1f872b None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:39.919499 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2596e56-bf15-4cb8-92e2-5bc00d1f872b None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:39.919789 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2596e56-bf15-4cb8-92e2-5bc00d1f872b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:39.985584 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689f2702-1c3a-49de-976d-19445e52303c None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:39.985874 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689f2702-1c3a-49de-976d-19445e52303c None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:39.986177 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689f2702-1c3a-49de-976d-19445e52303c None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:39.988295 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 702/1399] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:31:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 12 17:31:39.990038 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:39.993411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e866dfd7-be28-47b3-888f-e6be4fbee0fc None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:39.993704 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e866dfd7-be28-47b3-888f-e6be4fbee0fc None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:39.993991 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e866dfd7-be28-47b3-888f-e6be4fbee0fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:39.998241 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-c2596e56-bf15-4cb8-92e2-5bc00d1f872b None None] MFA Rules not processed for user `d7906d1a19c642bd93507a27bf3e3dae`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:31:40.004229 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-c2596e56-bf15-4cb8-92e2-5bc00d1f872b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:40.063747 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-e866dfd7-be28-47b3-888f-e6be4fbee0fc None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:31:40.065445 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-e866dfd7-be28-47b3-888f-e6be4fbee0fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:40.115578 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 698/1400] 10.222.0.16 () {58 vars in 1039 bytes} [Tue Jul 12 17:31:39 2022] POST /identity/v3/auth/tokens => generated 3294 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:31:40.124729 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2596e56-bf15-4cb8-92e2-5bc00d1f872b None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:31:40.134867 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2596e56-bf15-4cb8-92e2-5bc00d1f872b None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:31:40.177299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 703/1401] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:31:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:31:40.223621 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e866dfd7-be28-47b3-888f-e6be4fbee0fc None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:31:40.224892 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-e866dfd7-be28-47b3-888f-e6be4fbee0fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:40.247783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38627e8b-337a-42d2-a743-12e198b6ec08 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7906d1a19c642bd93507a27bf3e3dae', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:31:40.248441 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38627e8b-337a-42d2-a743-12e198b6ec08 service swift] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:40.248724 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38627e8b-337a-42d2-a743-12e198b6ec08 service swift] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:40.249007 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38627e8b-337a-42d2-a743-12e198b6ec08 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:40.250743 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-38627e8b-337a-42d2-a743-12e198b6ec08 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:40.252252 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:40.253548 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:40.337749 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38627e8b-337a-42d2-a743-12e198b6ec08 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:31:40.339379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38627e8b-337a-42d2-a743-12e198b6ec08 service swift] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:31:40.342349 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e866dfd7-be28-47b3-888f-e6be4fbee0fc None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:31:40.348445 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 699/1402] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Jul 12 17:31:40 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 224 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 12 17:31:40.461097 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfce5d49-ac25-4c89-96b6-0fd5de0772e3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:31:40.461694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfce5d49-ac25-4c89-96b6-0fd5de0772e3 service nova] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:40.461985 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfce5d49-ac25-4c89-96b6-0fd5de0772e3 service nova] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:40.462262 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfce5d49-ac25-4c89-96b6-0fd5de0772e3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:40.463982 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-dfce5d49-ac25-4c89-96b6-0fd5de0772e3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:40.481981 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:40.553361 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfce5d49-ac25-4c89-96b6-0fd5de0772e3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:31:40.555000 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfce5d49-ac25-4c89-96b6-0fd5de0772e3 service nova] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:31:40.574640 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 704/1403] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:31:40 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:31:42.805435 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119b6dc5-c809-41cb-a438-04ebd5f7a8f6 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:42.806232 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119b6dc5-c809-41cb-a438-04ebd5f7a8f6 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:42.806398 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119b6dc5-c809-41cb-a438-04ebd5f7a8f6 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:42.807967 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 700/1404] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:31:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:31:42.814884 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bf4ef9-c062-4828-99e2-b00d99f07291 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:42.814884 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bf4ef9-c062-4828-99e2-b00d99f07291 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:42.814884 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bf4ef9-c062-4828-99e2-b00d99f07291 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:42.891772 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-b2bf4ef9-c062-4828-99e2-b00d99f07291 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:31:42.893132 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-b2bf4ef9-c062-4828-99e2-b00d99f07291 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:43.005842 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:43.012811 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 705/1405] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:31:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:31:43.058825 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119b6dc5-c809-41cb-a438-04ebd5f7a8f6 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:31:43.061077 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-119b6dc5-c809-41cb-a438-04ebd5f7a8f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:43.257482 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:43.260209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119b6dc5-c809-41cb-a438-04ebd5f7a8f6 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:31:43.386919 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d3d48e-077f-4ea4-bc48-f34ed0721e1d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:31:43.387578 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d3d48e-077f-4ea4-bc48-f34ed0721e1d service nova] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:43.387873 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d3d48e-077f-4ea4-bc48-f34ed0721e1d service nova] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:43.388158 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d3d48e-077f-4ea4-bc48-f34ed0721e1d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:43.389831 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-b4d3d48e-077f-4ea4-bc48-f34ed0721e1d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:43.495122 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d3d48e-077f-4ea4-bc48-f34ed0721e1d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:31:43.496909 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d3d48e-077f-4ea4-bc48-f34ed0721e1d service nova] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:31:43.518319 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 701/1406] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:31:43 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 460 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:31:45.775804 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338267fa-3171-4ea7-8045-09b37f6c95ac None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:45.776325 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338267fa-3171-4ea7-8045-09b37f6c95ac None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:45.776325 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338267fa-3171-4ea7-8045-09b37f6c95ac None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:45.777663 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 706/1407] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:31:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:31:45.783418 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654085b4-91de-490d-8c82-a8c4d570bf4c None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:45.783919 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654085b4-91de-490d-8c82-a8c4d570bf4c None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:45.784142 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654085b4-91de-490d-8c82-a8c4d570bf4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:45.787442 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:45.858626 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-654085b4-91de-490d-8c82-a8c4d570bf4c None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:31:45.860133 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-654085b4-91de-490d-8c82-a8c4d570bf4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:45.974488 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 702/1408] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:31:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:31:46.017667 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338267fa-3171-4ea7-8045-09b37f6c95ac None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:31:46.019334 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-338267fa-3171-4ea7-8045-09b37f6c95ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:46.185610 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338267fa-3171-4ea7-8045-09b37f6c95ac None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:31:46.261951 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:46.314573 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097897b6-1bc4-463c-be99-53e8e5de7cff service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:31:46.315554 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097897b6-1bc4-463c-be99-53e8e5de7cff service nova] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:31:46.315834 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097897b6-1bc4-463c-be99-53e8e5de7cff service nova] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:31:46.316110 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097897b6-1bc4-463c-be99-53e8e5de7cff service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:31:46.317779 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-097897b6-1bc4-463c-be99-53e8e5de7cff service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:31:46.424974 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-097897b6-1bc4-463c-be99-53e8e5de7cff service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:31:46.426903 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-097897b6-1bc4-463c-be99-53e8e5de7cff service nova] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:31:46.450694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 707/1409] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:31:46 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 433 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:31:55.968174 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=64 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:31:56.444009 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=64 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:04.650778 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68091861-5bc9-427f-ad2b-fefce404969e None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:04.651376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68091861-5bc9-427f-ad2b-fefce404969e None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:04.651376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68091861-5bc9-427f-ad2b-fefce404969e None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:04.652727 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 703/1410] 10.222.0.16 () {54 vars in 918 bytes} [Tue Jul 12 17:32:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:32:04.658662 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39b4939-e432-45bb-b258-2dab5d70c6e3 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:04.659160 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39b4939-e432-45bb-b258-2dab5d70c6e3 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:04.659321 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39b4939-e432-45bb-b258-2dab5d70c6e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:04.734274 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-c39b4939-e432-45bb-b258-2dab5d70c6e3 None None] MFA Rules not processed for user `f623be28c83247c78d3ea6873c347fe0`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:32:04.736144 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-c39b4939-e432-45bb-b258-2dab5d70c6e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:04.841875 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 708/1411] 10.222.0.16 () {58 vars in 1015 bytes} [Tue Jul 12 17:32:04 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:32:04.857398 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3181e3d-dc3f-415e-83d4-7f92a2767548 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:04.857686 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3181e3d-dc3f-415e-83d4-7f92a2767548 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:04.857965 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3181e3d-dc3f-415e-83d4-7f92a2767548 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:04.859356 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 704/1412] 10.222.0.16 () {54 vars in 918 bytes} [Tue Jul 12 17:32:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:32:04.865750 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54041bdd-841d-4dda-89a2-6c9bc06d8dcc None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:04.866057 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54041bdd-841d-4dda-89a2-6c9bc06d8dcc None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:04.866351 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54041bdd-841d-4dda-89a2-6c9bc06d8dcc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:04.937427 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:04.940575 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-54041bdd-841d-4dda-89a2-6c9bc06d8dcc None None] MFA Rules not processed for user `9012d5b88f554dc596582a7693c2a72a`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:32:04.941801 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-54041bdd-841d-4dda-89a2-6c9bc06d8dcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:05.058961 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 709/1413] 10.222.0.16 () {58 vars in 1015 bytes} [Tue Jul 12 17:32:04 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:32:07.581003 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef5f5b2-cebd-434e-92dc-baac75cd247a None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:07.581419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef5f5b2-cebd-434e-92dc-baac75cd247a None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:07.581573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ef5f5b2-cebd-434e-92dc-baac75cd247a None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:07.583260 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 705/1414] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 12 17:32:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:32:07.588477 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32319000-32eb-48fc-bc55-e8d683fc75cd None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:07.588902 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32319000-32eb-48fc-bc55-e8d683fc75cd None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:07.589046 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32319000-32eb-48fc-bc55-e8d683fc75cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:07.655070 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-32319000-32eb-48fc-bc55-e8d683fc75cd None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:32:07.656302 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-32319000-32eb-48fc-bc55-e8d683fc75cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:07.712317 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:07.760193 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 710/1415] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 12 17:32:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:32:07.804052 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef5f5b2-cebd-434e-92dc-baac75cd247a None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:32:07.805619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-9ef5f5b2-cebd-434e-92dc-baac75cd247a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:07.954285 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ef5f5b2-cebd-434e-92dc-baac75cd247a None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:32:08.044574 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:08.088790 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb30bd6d-10ed-4fcf-a869-9fd75344a423 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:32:08.089420 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb30bd6d-10ed-4fcf-a869-9fd75344a423 service nova] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:08.089708 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb30bd6d-10ed-4fcf-a869-9fd75344a423 service nova] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:08.089992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb30bd6d-10ed-4fcf-a869-9fd75344a423 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:08.091787 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-eb30bd6d-10ed-4fcf-a869-9fd75344a423 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:08.185541 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb30bd6d-10ed-4fcf-a869-9fd75344a423 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:32:08.187219 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb30bd6d-10ed-4fcf-a869-9fd75344a423 service nova] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:32:08.207275 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 706/1416] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:32:07 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 404 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:32:09.128391 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048091c5-e0f9-4c01-aa25-345930b51d37 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:09.128909 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048091c5-e0f9-4c01-aa25-345930b51d37 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:09.129004 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-048091c5-e0f9-4c01-aa25-345930b51d37 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:09.130376 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 711/1417] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 12 17:32:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:32:09.135647 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-958d16f6-0989-4d8c-94f0-44ceb7a0768d None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:09.136093 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-958d16f6-0989-4d8c-94f0-44ceb7a0768d None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:09.136245 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-958d16f6-0989-4d8c-94f0-44ceb7a0768d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:09.215587 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-958d16f6-0989-4d8c-94f0-44ceb7a0768d None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:32:09.216804 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-958d16f6-0989-4d8c-94f0-44ceb7a0768d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:09.246130 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:09.323310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 707/1418] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 12 17:32:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:32:09.474911 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048091c5-e0f9-4c01-aa25-345930b51d37 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:32:09.476434 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-048091c5-e0f9-4c01-aa25-345930b51d37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:09.645026 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-048091c5-e0f9-4c01-aa25-345930b51d37 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:32:09.690391 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:09.774969 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e84506-00c5-47fa-980b-5f6ce068aecd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:32:09.775509 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e84506-00c5-47fa-980b-5f6ce068aecd service nova] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:09.775794 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e84506-00c5-47fa-980b-5f6ce068aecd service nova] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:09.776099 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e84506-00c5-47fa-980b-5f6ce068aecd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:09.777765 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-01e84506-00c5-47fa-980b-5f6ce068aecd service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:09.880408 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e84506-00c5-47fa-980b-5f6ce068aecd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:32:09.882195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e84506-00c5-47fa-980b-5f6ce068aecd service nova] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:32:09.903367 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 712/1419] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:32:09 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 429 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:32:10.823552 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0173780a-c068-4520-b739-b69d3d3de951 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:10.824099 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0173780a-c068-4520-b739-b69d3d3de951 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:10.824099 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0173780a-c068-4520-b739-b69d3d3de951 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:10.825482 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 708/1420] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 12 17:32:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:32:10.831403 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cdcf726-1803-4fbe-80eb-32575f5454c6 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:10.831844 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cdcf726-1803-4fbe-80eb-32575f5454c6 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:10.831940 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cdcf726-1803-4fbe-80eb-32575f5454c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:10.888163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:10.919886 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-5cdcf726-1803-4fbe-80eb-32575f5454c6 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:32:10.921890 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-5cdcf726-1803-4fbe-80eb-32575f5454c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:11.023496 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 713/1421] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 12 17:32:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:32:11.175446 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0173780a-c068-4520-b739-b69d3d3de951 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:32:11.177260 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-0173780a-c068-4520-b739-b69d3d3de951 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:11.336010 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0173780a-c068-4520-b739-b69d3d3de951 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:32:11.350244 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:11.459109 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f34688-996f-4ea1-a732-1c12d354aeb5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:32:11.459771 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f34688-996f-4ea1-a732-1c12d354aeb5 service nova] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:11.460055 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f34688-996f-4ea1-a732-1c12d354aeb5 service nova] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:11.460611 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f34688-996f-4ea1-a732-1c12d354aeb5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:11.462290 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-15f34688-996f-4ea1-a732-1c12d354aeb5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:11.558270 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f34688-996f-4ea1-a732-1c12d354aeb5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:32:11.559972 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f34688-996f-4ea1-a732-1c12d354aeb5 service nova] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:32:11.581853 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 709/1422] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:32:11 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 407 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:32:12.465457 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16940b09-f47a-40f7-92b3-83e5d629295e None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:12.466018 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16940b09-f47a-40f7-92b3-83e5d629295e None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:12.466018 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16940b09-f47a-40f7-92b3-83e5d629295e None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:12.467737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 714/1423] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 12 17:32:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:32:12.473633 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0087e0d-55d3-4bb2-a6e0-5217be31fa7f None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:12.474093 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0087e0d-55d3-4bb2-a6e0-5217be31fa7f None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:12.474193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0087e0d-55d3-4bb2-a6e0-5217be31fa7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:12.493813 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:12.553391 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-e0087e0d-55d3-4bb2-a6e0-5217be31fa7f None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:32:12.554644 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-e0087e0d-55d3-4bb2-a6e0-5217be31fa7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:12.671119 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 710/1424] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 12 17:32:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:32:12.825461 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16940b09-f47a-40f7-92b3-83e5d629295e None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:32:12.826936 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-16940b09-f47a-40f7-92b3-83e5d629295e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:12.951505 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:12.990337 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16940b09-f47a-40f7-92b3-83e5d629295e None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:32:13.121319 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e489aa5-b812-43b8-b43e-5f58b226a1c7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:32:13.121981 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e489aa5-b812-43b8-b43e-5f58b226a1c7 service nova] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:13.122269 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e489aa5-b812-43b8-b43e-5f58b226a1c7 service nova] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:13.122709 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e489aa5-b812-43b8-b43e-5f58b226a1c7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:13.124370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-9e489aa5-b812-43b8-b43e-5f58b226a1c7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:13.207540 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:13.221421 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e489aa5-b812-43b8-b43e-5f58b226a1c7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:32:13.223071 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e489aa5-b812-43b8-b43e-5f58b226a1c7 service nova] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:32:13.243497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 715/1425] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:32:12 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 419 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:32:14.123697 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200a16eb-6c06-4f54-9070-0d4a69c5d892 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:14.124235 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200a16eb-6c06-4f54-9070-0d4a69c5d892 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:14.124342 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-200a16eb-6c06-4f54-9070-0d4a69c5d892 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:14.126194 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 711/1426] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 12 17:32:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:32:14.131586 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8404874-4768-40aa-8e73-f8d33bb7ed9e None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:14.132062 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8404874-4768-40aa-8e73-f8d33bb7ed9e None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:14.132177 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8404874-4768-40aa-8e73-f8d33bb7ed9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:14.212204 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-a8404874-4768-40aa-8e73-f8d33bb7ed9e None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:32:14.213513 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-a8404874-4768-40aa-8e73-f8d33bb7ed9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:14.320400 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 716/1427] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 12 17:32:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:32:14.472212 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200a16eb-6c06-4f54-9070-0d4a69c5d892 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:32:14.473668 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-200a16eb-6c06-4f54-9070-0d4a69c5d892 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:14.563941 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:14.637808 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-200a16eb-6c06-4f54-9070-0d4a69c5d892 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:32:14.761973 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b27ea7d4-fc96-4e44-b59f-54aecbadd0f6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:32:14.762734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27ea7d4-fc96-4e44-b59f-54aecbadd0f6 service nova] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:14.763035 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27ea7d4-fc96-4e44-b59f-54aecbadd0f6 service nova] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:14.763347 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b27ea7d4-fc96-4e44-b59f-54aecbadd0f6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:14.765189 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-b27ea7d4-fc96-4e44-b59f-54aecbadd0f6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:14.814577 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:14.858361 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b27ea7d4-fc96-4e44-b59f-54aecbadd0f6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:32:14.860052 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b27ea7d4-fc96-4e44-b59f-54aecbadd0f6 service nova] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:32:14.878583 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 712/1428] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:32:14 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 407 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:32:15.762978 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a3e7e1-9804-4b7a-89c2-75ee773762c2 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:15.763727 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a3e7e1-9804-4b7a-89c2-75ee773762c2 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:15.763727 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a3e7e1-9804-4b7a-89c2-75ee773762c2 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:15.764842 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 717/1429] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 12 17:32:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:32:15.770755 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c27f06-9d53-49df-bbf2-adb47ee86ded None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:15.771199 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c27f06-9d53-49df-bbf2-adb47ee86ded None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:15.771398 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c27f06-9d53-49df-bbf2-adb47ee86ded None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:15.846222 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-09c27f06-9d53-49df-bbf2-adb47ee86ded None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:32:15.847703 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-09c27f06-9d53-49df-bbf2-adb47ee86ded None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:15.946103 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 713/1430] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 12 17:32:15 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:32:16.099836 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a3e7e1-9804-4b7a-89c2-75ee773762c2 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:32:16.101295 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-04a3e7e1-9804-4b7a-89c2-75ee773762c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:16.151436 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:16.252964 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a3e7e1-9804-4b7a-89c2-75ee773762c2 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:32:16.366137 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a78662b-17a4-4bf6-be33-75c747e3fb21 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:32:16.366737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a78662b-17a4-4bf6-be33-75c747e3fb21 service nova] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:16.367028 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a78662b-17a4-4bf6-be33-75c747e3fb21 service nova] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:16.367338 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a78662b-17a4-4bf6-be33-75c747e3fb21 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:16.369000 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-1a78662b-17a4-4bf6-be33-75c747e3fb21 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:16.376699 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:16.460083 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a78662b-17a4-4bf6-be33-75c747e3fb21 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:32:16.461694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a78662b-17a4-4bf6-be33-75c747e3fb21 service nova] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:32:16.480196 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 718/1431] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:32:16 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:32:17.443706 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698b6478-a12a-4019-9ada-7d90d3f80c3d None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:17.444223 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698b6478-a12a-4019-9ada-7d90d3f80c3d None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:17.444223 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698b6478-a12a-4019-9ada-7d90d3f80c3d None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:17.445569 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 714/1432] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 12 17:32:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:32:17.451005 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3577edb-0654-4973-a748-8dfd8401d9cb None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:17.451436 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3577edb-0654-4973-a748-8dfd8401d9cb None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:17.451591 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3577edb-0654-4973-a748-8dfd8401d9cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:17.532067 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-b3577edb-0654-4973-a748-8dfd8401d9cb None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:32:17.533291 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-b3577edb-0654-4973-a748-8dfd8401d9cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:17.608922 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:17.636822 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 719/1433] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 12 17:32:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:32:17.790874 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698b6478-a12a-4019-9ada-7d90d3f80c3d None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:32:17.792413 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-698b6478-a12a-4019-9ada-7d90d3f80c3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:17.809914 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:17.939818 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698b6478-a12a-4019-9ada-7d90d3f80c3d None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:32:18.028496 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:18.054538 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ba2a8b-f166-43ca-87cd-4a8f8e267176 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:32:18.055118 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ba2a8b-f166-43ca-87cd-4a8f8e267176 service nova] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:18.055412 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ba2a8b-f166-43ca-87cd-4a8f8e267176 service nova] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:18.055695 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ba2a8b-f166-43ca-87cd-4a8f8e267176 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:18.057361 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-55ba2a8b-f166-43ca-87cd-4a8f8e267176 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:18.148285 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ba2a8b-f166-43ca-87cd-4a8f8e267176 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:32:18.149905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ba2a8b-f166-43ca-87cd-4a8f8e267176 service nova] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:32:18.168224 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 715/1434] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:32:17 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:32:19.052867 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142a483e-4863-47c9-9704-584ae437b26d None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:19.053442 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142a483e-4863-47c9-9704-584ae437b26d None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:19.053442 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142a483e-4863-47c9-9704-584ae437b26d None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:19.054908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 720/1435] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 12 17:32:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:32:19.060076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcece17d-f03f-431b-bf3c-86b1f26b5b21 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:19.060491 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcece17d-f03f-431b-bf3c-86b1f26b5b21 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:19.060653 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcece17d-f03f-431b-bf3c-86b1f26b5b21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:19.136354 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-bcece17d-f03f-431b-bf3c-86b1f26b5b21 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:32:19.137574 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-bcece17d-f03f-431b-bf3c-86b1f26b5b21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:19.181414 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:19.239176 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 716/1436] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 12 17:32:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:32:19.426747 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142a483e-4863-47c9-9704-584ae437b26d None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:32:19.426747 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-142a483e-4863-47c9-9704-584ae437b26d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:19.553022 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142a483e-4863-47c9-9704-584ae437b26d None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:32:19.607942 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:19.668419 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e5c31e-8623-471e-a5de-69a00b5ad4ba service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:32:19.669071 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e5c31e-8623-471e-a5de-69a00b5ad4ba service nova] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:19.669359 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e5c31e-8623-471e-a5de-69a00b5ad4ba service nova] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:19.669641 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e5c31e-8623-471e-a5de-69a00b5ad4ba service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:19.671331 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-30e5c31e-8623-471e-a5de-69a00b5ad4ba service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:19.762193 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e5c31e-8623-471e-a5de-69a00b5ad4ba service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:32:19.763848 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e5c31e-8623-471e-a5de-69a00b5ad4ba service nova] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:32:19.782225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 721/1437] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:32:19 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 385 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:32:20.657862 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79451b2-ef63-4a15-95f7-e8d1a8023a56 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:20.658475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79451b2-ef63-4a15-95f7-e8d1a8023a56 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:20.658475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79451b2-ef63-4a15-95f7-e8d1a8023a56 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:20.659835 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 717/1438] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 12 17:32:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:32:20.664826 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b38217-9652-4e78-9bd9-f2f9d310e7f9 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:20.665264 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b38217-9652-4e78-9bd9-f2f9d310e7f9 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:20.665372 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b38217-9652-4e78-9bd9-f2f9d310e7f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:20.741311 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-74b38217-9652-4e78-9bd9-f2f9d310e7f9 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:32:20.742593 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-74b38217-9652-4e78-9bd9-f2f9d310e7f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:20.744621 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:20.839658 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 722/1439] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 12 17:32:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:32:20.991372 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f79451b2-ef63-4a15-95f7-e8d1a8023a56 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:32:20.992935 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-f79451b2-ef63-4a15-95f7-e8d1a8023a56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:21.139439 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f79451b2-ef63-4a15-95f7-e8d1a8023a56 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:32:21.166833 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:21.253926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0171c0-b95a-41c2-94ba-8bcdaed319fa service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:32:21.254624 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0171c0-b95a-41c2-94ba-8bcdaed319fa service nova] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:21.254915 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0171c0-b95a-41c2-94ba-8bcdaed319fa service nova] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:21.255197 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0171c0-b95a-41c2-94ba-8bcdaed319fa service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:21.256867 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-df0171c0-b95a-41c2-94ba-8bcdaed319fa service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:21.346096 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df0171c0-b95a-41c2-94ba-8bcdaed319fa service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:32:21.347645 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df0171c0-b95a-41c2-94ba-8bcdaed319fa service nova] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:32:21.365532 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 718/1440] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:32:20 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:32:23.308687 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0fc0ff-d421-48ea-a736-de90383ebdc8 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:23.309258 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0fc0ff-d421-48ea-a736-de90383ebdc8 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:23.309258 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0fc0ff-d421-48ea-a736-de90383ebdc8 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:23.310656 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 723/1441] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 12 17:32:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:32:23.316512 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1090925-eb48-445e-a7fc-b1039eb20d69 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:23.316950 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1090925-eb48-445e-a7fc-b1039eb20d69 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:23.317103 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1090925-eb48-445e-a7fc-b1039eb20d69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:23.356903 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:23.382112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-f1090925-eb48-445e-a7fc-b1039eb20d69 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:32:23.383346 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-f1090925-eb48-445e-a7fc-b1039eb20d69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:23.479879 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 719/1442] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 12 17:32:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:32:23.518894 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0fc0ff-d421-48ea-a736-de90383ebdc8 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:32:23.520441 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-7c0fc0ff-d421-48ea-a736-de90383ebdc8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:23.626724 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0fc0ff-d421-48ea-a736-de90383ebdc8 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:32:23.659020 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:23.701871 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e836c3-de80-48cc-a693-79c394a86793 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:32:23.702607 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e836c3-de80-48cc-a693-79c394a86793 service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:23.702898 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e836c3-de80-48cc-a693-79c394a86793 service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:23.703207 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e836c3-de80-48cc-a693-79c394a86793 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:23.704875 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-a1e836c3-de80-48cc-a693-79c394a86793 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:23.817049 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e836c3-de80-48cc-a693-79c394a86793 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:32:23.818531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e836c3-de80-48cc-a693-79c394a86793 service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:32:23.837514 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 724/1443] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:32:23 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:32:26.049345 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea9edef-c9ad-43ad-9107-6be7c6ee3499 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:26.049923 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea9edef-c9ad-43ad-9107-6be7c6ee3499 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:26.049923 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea9edef-c9ad-43ad-9107-6be7c6ee3499 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:26.051332 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 720/1444] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 12 17:32:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:32:26.056915 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b2fa5d-f892-4554-be47-f800263ed72f None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:26.057349 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b2fa5d-f892-4554-be47-f800263ed72f None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:26.057503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b2fa5d-f892-4554-be47-f800263ed72f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:26.129388 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-91b2fa5d-f892-4554-be47-f800263ed72f None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:32:26.130654 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-91b2fa5d-f892-4554-be47-f800263ed72f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:26.143730 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:26.227137 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 725/1445] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 12 17:32:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:32:26.265276 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea9edef-c9ad-43ad-9107-6be7c6ee3499 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:32:26.267495 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-aea9edef-c9ad-43ad-9107-6be7c6ee3499 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:26.372108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea9edef-c9ad-43ad-9107-6be7c6ee3499 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:32:26.384768 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:26.446951 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d564fb1d-d457-49b5-a6f1-5337b481efb1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:32:26.447602 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d564fb1d-d457-49b5-a6f1-5337b481efb1 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:26.447890 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d564fb1d-d457-49b5-a6f1-5337b481efb1 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:26.448223 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d564fb1d-d457-49b5-a6f1-5337b481efb1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:26.449817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-d564fb1d-d457-49b5-a6f1-5337b481efb1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:26.559531 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d564fb1d-d457-49b5-a6f1-5337b481efb1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:32:26.560968 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d564fb1d-d457-49b5-a6f1-5337b481efb1 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:32:26.580477 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 721/1446] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:32:26 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:32:28.991842 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba03a9a-ed09-46fa-9adf-2443e1c2ed74 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:28.992403 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba03a9a-ed09-46fa-9adf-2443e1c2ed74 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:28.992403 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba03a9a-ed09-46fa-9adf-2443e1c2ed74 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:28.993843 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 726/1447] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 12 17:32:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:32:28.999127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f11316-bda3-4ee6-adb1-d77c06dd3464 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:28.999595 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f11316-bda3-4ee6-adb1-d77c06dd3464 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:28.999777 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f11316-bda3-4ee6-adb1-d77c06dd3464 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:29.046996 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:29.065919 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-c5f11316-bda3-4ee6-adb1-d77c06dd3464 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:32:29.067193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-c5f11316-bda3-4ee6-adb1-d77c06dd3464 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:29.168782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 722/1448] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 12 17:32:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:32:29.211837 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba03a9a-ed09-46fa-9adf-2443e1c2ed74 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:32:29.213418 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-3ba03a9a-ed09-46fa-9adf-2443e1c2ed74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:29.327104 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba03a9a-ed09-46fa-9adf-2443e1c2ed74 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:32:29.378573 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:29.415960 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb03567-bad2-4682-8774-c35431c1e15f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:32:29.416739 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb03567-bad2-4682-8774-c35431c1e15f service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:29.417134 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb03567-bad2-4682-8774-c35431c1e15f service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:29.417426 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb03567-bad2-4682-8774-c35431c1e15f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:29.419155 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-0eb03567-bad2-4682-8774-c35431c1e15f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:29.547911 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eb03567-bad2-4682-8774-c35431c1e15f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:32:29.550108 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eb03567-bad2-4682-8774-c35431c1e15f service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:32:29.580709 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 727/1449] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:32:29 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:32:31.853904 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847333ab-56e4-4d78-bc6a-458363ffcff9 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:31.854580 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847333ab-56e4-4d78-bc6a-458363ffcff9 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:31.854580 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847333ab-56e4-4d78-bc6a-458363ffcff9 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:31.855799 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 723/1450] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 12 17:32:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:32:31.861053 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1362b631-45ad-45d9-abb8-ff9285c27501 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:31.861500 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1362b631-45ad-45d9-abb8-ff9285c27501 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:31.861612 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1362b631-45ad-45d9-abb8-ff9285c27501 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:31.926288 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-1362b631-45ad-45d9-abb8-ff9285c27501 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:32:31.927547 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-1362b631-45ad-45d9-abb8-ff9285c27501 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:31.947849 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:32.025822 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 728/1451] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 12 17:32:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:32:32.064521 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847333ab-56e4-4d78-bc6a-458363ffcff9 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:32:32.066186 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-847333ab-56e4-4d78-bc6a-458363ffcff9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:32.170382 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847333ab-56e4-4d78-bc6a-458363ffcff9 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:32:32.191783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:32.247608 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0251d4cb-cb78-4bf1-88a2-f385cbe9629b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:32:32.248261 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0251d4cb-cb78-4bf1-88a2-f385cbe9629b service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:32.248550 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0251d4cb-cb78-4bf1-88a2-f385cbe9629b service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:32.248833 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0251d4cb-cb78-4bf1-88a2-f385cbe9629b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:32.251201 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-0251d4cb-cb78-4bf1-88a2-f385cbe9629b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:32.359246 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0251d4cb-cb78-4bf1-88a2-f385cbe9629b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:32:32.360679 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0251d4cb-cb78-4bf1-88a2-f385cbe9629b service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:32:32.379193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 724/1452] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:32:32 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:32:34.467996 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0173b6-7169-4057-b878-e84de0a256eb None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:34.469037 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0173b6-7169-4057-b878-e84de0a256eb None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:34.469037 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0173b6-7169-4057-b878-e84de0a256eb None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:34.470655 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 729/1453] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 12 17:32:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:32:34.475916 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9bfed76-82fb-4354-9851-e4eee6a13181 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:34.476354 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9bfed76-82fb-4354-9851-e4eee6a13181 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:34.476493 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9bfed76-82fb-4354-9851-e4eee6a13181 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:34.539237 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:34.541710 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-c9bfed76-82fb-4354-9851-e4eee6a13181 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:32:34.543058 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-c9bfed76-82fb-4354-9851-e4eee6a13181 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:34.650146 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 725/1454] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 12 17:32:34 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:32:34.690642 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0173b6-7169-4057-b878-e84de0a256eb None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:32:34.692184 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-2c0173b6-7169-4057-b878-e84de0a256eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:34.799618 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0173b6-7169-4057-b878-e84de0a256eb None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:32:34.848310 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:34.873163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6b3738f-edb1-49c0-8e18-021a40f8f86b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:32:34.873817 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b3738f-edb1-49c0-8e18-021a40f8f86b service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:34.874103 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b3738f-edb1-49c0-8e18-021a40f8f86b service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:34.875040 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b3738f-edb1-49c0-8e18-021a40f8f86b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:34.877336 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-d6b3738f-edb1-49c0-8e18-021a40f8f86b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:34.984274 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6b3738f-edb1-49c0-8e18-021a40f8f86b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:32:34.985710 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6b3738f-edb1-49c0-8e18-021a40f8f86b service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:32:35.004490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 730/1455] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:32:34 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:32:37.093121 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2eddcdf-ab68-46c7-85db-4764cf94fa4a None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:37.093687 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2eddcdf-ab68-46c7-85db-4764cf94fa4a None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:37.093796 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2eddcdf-ab68-46c7-85db-4764cf94fa4a None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:37.095179 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 726/1456] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 12 17:32:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:32:37.100548 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c36fc00-8e23-414b-b7e7-d78700ecf7ab None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:37.100966 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c36fc00-8e23-414b-b7e7-d78700ecf7ab None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:37.101118 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c36fc00-8e23-414b-b7e7-d78700ecf7ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:37.166209 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-6c36fc00-8e23-414b-b7e7-d78700ecf7ab None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:32:37.167463 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-6c36fc00-8e23-414b-b7e7-d78700ecf7ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:37.197602 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:37.266964 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 731/1457] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 12 17:32:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:32:37.305065 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2eddcdf-ab68-46c7-85db-4764cf94fa4a None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:32:37.306647 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-b2eddcdf-ab68-46c7-85db-4764cf94fa4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:37.409443 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2eddcdf-ab68-46c7-85db-4764cf94fa4a None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:32:37.437553 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:37.483305 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd1a8d9-3974-4a46-b5b9-d9b2c2536331 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:32:37.485017 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd1a8d9-3974-4a46-b5b9-d9b2c2536331 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:37.485436 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd1a8d9-3974-4a46-b5b9-d9b2c2536331 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:37.485722 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd1a8d9-3974-4a46-b5b9-d9b2c2536331 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:37.487456 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-ebd1a8d9-3974-4a46-b5b9-d9b2c2536331 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:37.599747 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd1a8d9-3974-4a46-b5b9-d9b2c2536331 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:32:37.601193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd1a8d9-3974-4a46-b5b9-d9b2c2536331 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:32:37.621026 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 727/1458] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:32:37 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:32:39.786017 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f37a19-276f-4eca-9a8f-cdcb780cc15d None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:39.786673 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f37a19-276f-4eca-9a8f-cdcb780cc15d None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:39.786673 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f37a19-276f-4eca-9a8f-cdcb780cc15d None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:39.788082 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 732/1459] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 12 17:32:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:32:39.793235 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ee648b-8e5d-4e7a-bdb3-bc123dbeac30 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:39.793642 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ee648b-8e5d-4e7a-bdb3-bc123dbeac30 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:39.793786 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ee648b-8e5d-4e7a-bdb3-bc123dbeac30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:39.858060 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-47ee648b-8e5d-4e7a-bdb3-bc123dbeac30 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:32:39.859272 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-47ee648b-8e5d-4e7a-bdb3-bc123dbeac30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:39.867401 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:39.959178 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 728/1460] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 12 17:32:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:32:39.997104 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f37a19-276f-4eca-9a8f-cdcb780cc15d None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:32:39.998693 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-b2f37a19-276f-4eca-9a8f-cdcb780cc15d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:40.105908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f37a19-276f-4eca-9a8f-cdcb780cc15d None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:32:40.163137 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:40.182692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b32aca-eddf-4aa6-bc64-a3f26e151561 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:32:40.183346 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b32aca-eddf-4aa6-bc64-a3f26e151561 service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:40.183673 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b32aca-eddf-4aa6-bc64-a3f26e151561 service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:40.183949 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b32aca-eddf-4aa6-bc64-a3f26e151561 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:40.185612 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-09b32aca-eddf-4aa6-bc64-a3f26e151561 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:40.298322 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09b32aca-eddf-4aa6-bc64-a3f26e151561 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:32:40.299801 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09b32aca-eddf-4aa6-bc64-a3f26e151561 service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:32:40.319138 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 733/1461] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:32:39 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:32:42.430498 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221ec1ce-71b7-476c-906c-152ef347a086 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:42.431076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221ec1ce-71b7-476c-906c-152ef347a086 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:42.431076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221ec1ce-71b7-476c-906c-152ef347a086 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:42.432451 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 729/1462] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 12 17:32:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:32:42.437683 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fa9340-a7bb-4e7f-9abf-6ccbb4c21a2d None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:42.438164 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fa9340-a7bb-4e7f-9abf-6ccbb4c21a2d None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:42.438276 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7fa9340-a7bb-4e7f-9abf-6ccbb4c21a2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:42.507036 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-c7fa9340-a7bb-4e7f-9abf-6ccbb4c21a2d None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:32:42.508246 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-c7fa9340-a7bb-4e7f-9abf-6ccbb4c21a2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:42.547985 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:42.609591 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 734/1463] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 12 17:32:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:32:42.647163 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221ec1ce-71b7-476c-906c-152ef347a086 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:32:42.649495 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-221ec1ce-71b7-476c-906c-152ef347a086 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:42.771554 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221ec1ce-71b7-476c-906c-152ef347a086 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:32:42.812271 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:42.850575 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d2e101-db07-4451-b8ec-02c198449bbc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:32:42.851240 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d2e101-db07-4451-b8ec-02c198449bbc service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:42.851534 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d2e101-db07-4451-b8ec-02c198449bbc service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:42.851821 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d2e101-db07-4451-b8ec-02c198449bbc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:42.853492 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-95d2e101-db07-4451-b8ec-02c198449bbc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:42.968601 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95d2e101-db07-4451-b8ec-02c198449bbc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:32:42.970055 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95d2e101-db07-4451-b8ec-02c198449bbc service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:32:42.989868 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 730/1464] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:32:42 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:32:45.075580 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f81336-02bb-48eb-84d3-2f4be0f688e7 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:45.076121 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f81336-02bb-48eb-84d3-2f4be0f688e7 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:45.076121 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f81336-02bb-48eb-84d3-2f4be0f688e7 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:45.078367 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 735/1465] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 12 17:32:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:32:45.083649 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2349c247-e8ed-4cf0-89f7-1c597792f482 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:45.084063 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2349c247-e8ed-4cf0-89f7-1c597792f482 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:45.084211 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2349c247-e8ed-4cf0-89f7-1c597792f482 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:45.153189 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-2349c247-e8ed-4cf0-89f7-1c597792f482 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:32:45.154472 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-2349c247-e8ed-4cf0-89f7-1c597792f482 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:45.173149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:45.254677 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 731/1466] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 12 17:32:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:32:45.291465 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f81336-02bb-48eb-84d3-2f4be0f688e7 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:32:45.293026 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-c5f81336-02bb-48eb-84d3-2f4be0f688e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:45.400611 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f81336-02bb-48eb-84d3-2f4be0f688e7 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:32:45.468574 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:45.475624 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d47b33b-d95f-4340-8c93-ef8cbe0f7e1e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:32:45.478916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d47b33b-d95f-4340-8c93-ef8cbe0f7e1e service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:45.478916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d47b33b-d95f-4340-8c93-ef8cbe0f7e1e service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:45.478916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d47b33b-d95f-4340-8c93-ef8cbe0f7e1e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:45.478916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-5d47b33b-d95f-4340-8c93-ef8cbe0f7e1e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:45.593011 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d47b33b-d95f-4340-8c93-ef8cbe0f7e1e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:32:45.594479 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d47b33b-d95f-4340-8c93-ef8cbe0f7e1e service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:32:45.612545 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 736/1467] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:32:45 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:32:47.673760 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2237efb-e387-41b9-9162-91420f0fc948 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:47.674321 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2237efb-e387-41b9-9162-91420f0fc948 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:47.674321 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2237efb-e387-41b9-9162-91420f0fc948 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:47.675766 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 732/1468] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 12 17:32:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:32:47.681823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03602a4-bbb4-480c-9242-494dde8d3374 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:47.682315 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03602a4-bbb4-480c-9242-494dde8d3374 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:47.682657 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03602a4-bbb4-480c-9242-494dde8d3374 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:47.755596 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-d03602a4-bbb4-480c-9242-494dde8d3374 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:32:47.756879 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-d03602a4-bbb4-480c-9242-494dde8d3374 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:47.808654 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:47.861727 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 737/1469] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 12 17:32:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:32:47.898714 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2237efb-e387-41b9-9162-91420f0fc948 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:32:47.900272 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-e2237efb-e387-41b9-9162-91420f0fc948 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:48.010139 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2237efb-e387-41b9-9162-91420f0fc948 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:32:48.060108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:48.087175 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e15dae5-2cc0-46cc-99bf-9b6a94b5db64 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:32:48.087852 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e15dae5-2cc0-46cc-99bf-9b6a94b5db64 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:32:48.088358 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e15dae5-2cc0-46cc-99bf-9b6a94b5db64 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:32:48.088701 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e15dae5-2cc0-46cc-99bf-9b6a94b5db64 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:32:48.090976 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-2e15dae5-2cc0-46cc-99bf-9b6a94b5db64 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:32:48.204139 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e15dae5-2cc0-46cc-99bf-9b6a94b5db64 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:32:48.205495 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e15dae5-2cc0-46cc-99bf-9b6a94b5db64 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:32:48.226370 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 733/1470] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:32:47 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:32:57.870326 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=22 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:32:58.223968 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=66 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:33:08.457540 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a831298b-bd1c-4f04-a459-870b45cd2fb4 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:33:08.458138 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a831298b-bd1c-4f04-a459-870b45cd2fb4 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:33:08.458138 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a831298b-bd1c-4f04-a459-870b45cd2fb4 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:33:08.459530 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 738/1471] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:33:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:33:08.465481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58eeb8b-5382-418d-bac4-e5fd32db58a0 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:33:08.465917 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58eeb8b-5382-418d-bac4-e5fd32db58a0 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:33:08.466030 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58eeb8b-5382-418d-bac4-e5fd32db58a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:33:08.531749 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-e58eeb8b-5382-418d-bac4-e5fd32db58a0 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:33:08.532976 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-e58eeb8b-5382-418d-bac4-e5fd32db58a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:33:08.635376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 734/1472] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:33:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:33:08.725490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc299700-377d-49b5-9a32-05c789b5ca82 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:33:08.725779 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc299700-377d-49b5-9a32-05c789b5ca82 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:33:08.726059 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc299700-377d-49b5-9a32-05c789b5ca82 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:33:08.727495 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 739/1473] 10.222.0.16 () {54 vars in 962 bytes} [Tue Jul 12 17:33:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:33:08.733958 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595ead7e-44ed-42e2-817c-6ed747475287 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:33:08.734248 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595ead7e-44ed-42e2-817c-6ed747475287 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:33:08.734560 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595ead7e-44ed-42e2-817c-6ed747475287 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:33:08.813554 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-595ead7e-44ed-42e2-817c-6ed747475287 None None] MFA Rules not processed for user `4018d94a1feb4d6dade88ee9663f9729`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:33:08.814968 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-595ead7e-44ed-42e2-817c-6ed747475287 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:33:08.836720 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:33:08.896114 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 735/1474] 10.222.0.16 () {58 vars in 1059 bytes} [Tue Jul 12 17:33:08 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:33:08.903811 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc299700-377d-49b5-9a32-05c789b5ca82 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:33:08.905850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-cc299700-377d-49b5-9a32-05c789b5ca82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:33:09.013192 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc299700-377d-49b5-9a32-05c789b5ca82 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:33:09.087085 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ccab496-edd7-436d-a022-64d3fe507a37 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4018d94a1feb4d6dade88ee9663f9729', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:33:09.087742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccab496-edd7-436d-a022-64d3fe507a37 service glance] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:33:09.088031 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccab496-edd7-436d-a022-64d3fe507a37 service glance] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:33:09.088311 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccab496-edd7-436d-a022-64d3fe507a37 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:33:09.090252 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-3ccab496-edd7-436d-a022-64d3fe507a37 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:33:09.146982 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:33:09.204595 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ccab496-edd7-436d-a022-64d3fe507a37 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:33:09.206043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ccab496-edd7-436d-a022-64d3fe507a37 service glance] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:33:09.225292 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 740/1475] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:33:08 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:33:11.400380 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440d146f-f6d2-4de0-8cac-5bda3aa1431b None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:33:11.400934 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440d146f-f6d2-4de0-8cac-5bda3aa1431b None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:33:11.400934 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-440d146f-f6d2-4de0-8cac-5bda3aa1431b None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:33:11.402238 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 736/1476] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:33:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:33:11.408222 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dfd788-2719-4d5a-8ed8-0d215ebbe99a None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:33:11.408633 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dfd788-2719-4d5a-8ed8-0d215ebbe99a None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:33:11.408793 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dfd788-2719-4d5a-8ed8-0d215ebbe99a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:33:11.473031 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-16dfd788-2719-4d5a-8ed8-0d215ebbe99a None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:33:11.474259 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-16dfd788-2719-4d5a-8ed8-0d215ebbe99a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:33:11.570725 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 741/1477] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:33:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:33:11.596063 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440d146f-f6d2-4de0-8cac-5bda3aa1431b None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:33:11.597602 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-440d146f-f6d2-4de0-8cac-5bda3aa1431b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:33:11.725082 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-440d146f-f6d2-4de0-8cac-5bda3aa1431b None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:33:11.774983 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:33:11.937749 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffca496-6097-418a-bc94-97806988859d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:33:11.938410 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffca496-6097-418a-bc94-97806988859d admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:33:11.938734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffca496-6097-418a-bc94-97806988859d admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:33:11.939018 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffca496-6097-418a-bc94-97806988859d admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:33:11.939761 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ffca496-6097-418a-bc94-97806988859d admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:33:11.941472 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: WARNING py.warnings [None req-8ffca496-6097-418a-bc94-97806988859d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 12 17:33:11.941472 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: warnings.warn(msg) Jul 12 17:33:11.941472 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: Jul 12 17:33:11.942641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ffca496-6097-418a-bc94-97806988859d admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:33:11.962402 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 737/1478] 10.222.0.16 () {56 vars in 1108 bytes} [Tue Jul 12 17:33:11 2022] GET /identity/v3/projects => generated 2553 bytes in 367 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:33:14.051685 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278ba2bf-00d0-4b71-bdc1-4cb8df29a3df None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:33:14.052271 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278ba2bf-00d0-4b71-bdc1-4cb8df29a3df None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:33:14.052271 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278ba2bf-00d0-4b71-bdc1-4cb8df29a3df None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:33:14.053706 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 742/1479] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:33:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:33:14.059202 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fecc717-6acc-4e09-83ff-752d57ffec01 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:33:14.059638 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fecc717-6acc-4e09-83ff-752d57ffec01 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:33:14.059795 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fecc717-6acc-4e09-83ff-752d57ffec01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:33:14.125609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-8fecc717-6acc-4e09-83ff-752d57ffec01 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:33:14.126916 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-8fecc717-6acc-4e09-83ff-752d57ffec01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:33:14.225485 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 738/1480] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:33:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:33:14.269679 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-278ba2bf-00d0-4b71-bdc1-4cb8df29a3df None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:33:14.271238 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-278ba2bf-00d0-4b71-bdc1-4cb8df29a3df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:33:14.281884 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:33:14.421847 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-278ba2bf-00d0-4b71-bdc1-4cb8df29a3df None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:33:14.500650 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:33:14.533734 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a9a2b5-499e-4f61-8b03-5f7725c25dfa service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:33:14.534309 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a9a2b5-499e-4f61-8b03-5f7725c25dfa service nova] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:33:14.534618 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a9a2b5-499e-4f61-8b03-5f7725c25dfa service nova] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:33:14.534897 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a9a2b5-499e-4f61-8b03-5f7725c25dfa service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:33:14.536531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-d0a9a2b5-499e-4f61-8b03-5f7725c25dfa service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:33:14.625787 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a9a2b5-499e-4f61-8b03-5f7725c25dfa service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:33:14.627500 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a9a2b5-499e-4f61-8b03-5f7725c25dfa service nova] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:33:14.647123 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 743/1481] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:33:14 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:33:16.850520 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627f0c20-5536-4da6-b601-b48d0c2f6e84 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:33:16.851076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627f0c20-5536-4da6-b601-b48d0c2f6e84 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:33:16.851076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627f0c20-5536-4da6-b601-b48d0c2f6e84 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:33:16.852364 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 739/1482] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:33:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:33:16.858216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d4b730-d452-4191-bb5c-a4305a8e42f4 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:33:16.858662 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d4b730-d452-4191-bb5c-a4305a8e42f4 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:33:16.858821 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d4b730-d452-4191-bb5c-a4305a8e42f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:33:16.925820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-71d4b730-d452-4191-bb5c-a4305a8e42f4 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:33:16.927073 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-71d4b730-d452-4191-bb5c-a4305a8e42f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:33:16.974874 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:33:17.026522 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 744/1483] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:33:16 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:33:17.062336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627f0c20-5536-4da6-b601-b48d0c2f6e84 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:33:17.063942 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-627f0c20-5536-4da6-b601-b48d0c2f6e84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:33:17.088396 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:33:17.170218 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627f0c20-5536-4da6-b601-b48d0c2f6e84 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:33:17.248224 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0bc96bd-d117-454a-8c13-938c4456a402 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:33:17.248990 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bc96bd-d117-454a-8c13-938c4456a402 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:33:17.249285 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bc96bd-d117-454a-8c13-938c4456a402 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:33:17.249573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bc96bd-d117-454a-8c13-938c4456a402 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:33:17.251305 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-f0bc96bd-d117-454a-8c13-938c4456a402 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:33:17.328026 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:33:17.364491 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0bc96bd-d117-454a-8c13-938c4456a402 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:33:17.365927 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0bc96bd-d117-454a-8c13-938c4456a402 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:33:17.384663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 740/1484] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:33:17 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:33:19.506331 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c094a2f1-a49d-4fd2-995e-cd892d410f63 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:33:19.506898 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c094a2f1-a49d-4fd2-995e-cd892d410f63 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:33:19.506898 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c094a2f1-a49d-4fd2-995e-cd892d410f63 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:33:19.508320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 745/1485] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:33:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:33:19.513485 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a759fd6-cd45-407d-b3e6-a83d205e3e52 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:33:19.513898 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a759fd6-cd45-407d-b3e6-a83d205e3e52 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:33:19.514044 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a759fd6-cd45-407d-b3e6-a83d205e3e52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:33:19.580578 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-8a759fd6-cd45-407d-b3e6-a83d205e3e52 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:33:19.581856 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-8a759fd6-cd45-407d-b3e6-a83d205e3e52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:33:19.680515 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 741/1486] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:33:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:33:19.719148 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c094a2f1-a49d-4fd2-995e-cd892d410f63 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:33:19.720705 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-c094a2f1-a49d-4fd2-995e-cd892d410f63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:33:19.826974 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c094a2f1-a49d-4fd2-995e-cd892d410f63 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:33:19.898443 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:33:19.900332 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c0a305-50f2-4db7-84c8-6e4247c2e99b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:33:19.900982 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c0a305-50f2-4db7-84c8-6e4247c2e99b service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:33:19.901272 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c0a305-50f2-4db7-84c8-6e4247c2e99b service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:33:19.901551 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c0a305-50f2-4db7-84c8-6e4247c2e99b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:33:19.903354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-30c0a305-50f2-4db7-84c8-6e4247c2e99b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:33:20.013526 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c0a305-50f2-4db7-84c8-6e4247c2e99b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:33:20.014995 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c0a305-50f2-4db7-84c8-6e4247c2e99b service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:33:20.033054 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 746/1487] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:33:19 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:33:22.327856 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bb537c-e38f-4b29-8e85-49ddb03c385e None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:33:22.328466 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bb537c-e38f-4b29-8e85-49ddb03c385e None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:33:22.328466 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bb537c-e38f-4b29-8e85-49ddb03c385e None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:33:22.329841 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 742/1488] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:33:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:33:22.335040 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a443c5-85bd-4a05-a082-0a836b82ac34 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:33:22.335457 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a443c5-85bd-4a05-a082-0a836b82ac34 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:33:22.335607 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a443c5-85bd-4a05-a082-0a836b82ac34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:33:22.400469 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-88a443c5-85bd-4a05-a082-0a836b82ac34 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:33:22.401700 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-88a443c5-85bd-4a05-a082-0a836b82ac34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:33:22.454483 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:33:22.498648 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 747/1489] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:33:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:33:22.539646 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bb537c-e38f-4b29-8e85-49ddb03c385e None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:33:22.541186 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-13bb537c-e38f-4b29-8e85-49ddb03c385e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:33:22.579980 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:33:22.649940 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bb537c-e38f-4b29-8e85-49ddb03c385e None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:33:22.725221 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6151b206-1e11-49df-9513-9c8a8cb1e540 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:33:22.725871 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6151b206-1e11-49df-9513-9c8a8cb1e540 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:33:22.726161 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6151b206-1e11-49df-9513-9c8a8cb1e540 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:33:22.726458 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6151b206-1e11-49df-9513-9c8a8cb1e540 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:33:22.728136 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-6151b206-1e11-49df-9513-9c8a8cb1e540 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:33:22.812134 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:33:22.840930 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6151b206-1e11-49df-9513-9c8a8cb1e540 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:33:22.842393 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6151b206-1e11-49df-9513-9c8a8cb1e540 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:33:22.861875 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 743/1490] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:33:22 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:33:25.380515 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e730d60c-3b8d-4e7e-8d28-cd0b64f03831 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:33:25.381060 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e730d60c-3b8d-4e7e-8d28-cd0b64f03831 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:33:25.381060 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e730d60c-3b8d-4e7e-8d28-cd0b64f03831 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:33:25.382371 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 748/1491] 10.222.0.16 () {54 vars in 926 bytes} [Tue Jul 12 17:33:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:33:25.388172 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc9dac6-0cc2-4479-8557-f170638a9d28 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:33:25.388621 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc9dac6-0cc2-4479-8557-f170638a9d28 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:33:25.388743 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc9dac6-0cc2-4479-8557-f170638a9d28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:33:25.453762 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-0dc9dac6-0cc2-4479-8557-f170638a9d28 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:33:25.455043 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-0dc9dac6-0cc2-4479-8557-f170638a9d28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:33:25.552036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 744/1492] 10.222.0.16 () {58 vars in 1023 bytes} [Tue Jul 12 17:33:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:33:25.592039 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e730d60c-3b8d-4e7e-8d28-cd0b64f03831 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:33:25.593584 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-e730d60c-3b8d-4e7e-8d28-cd0b64f03831 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:33:25.698402 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e730d60c-3b8d-4e7e-8d28-cd0b64f03831 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:33:25.773878 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dccfb33c-35e4-45a0-98ea-690bc1963013 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:33:25.774581 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dccfb33c-35e4-45a0-98ea-690bc1963013 service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:33:25.774875 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dccfb33c-35e4-45a0-98ea-690bc1963013 service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:33:25.775161 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dccfb33c-35e4-45a0-98ea-690bc1963013 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:33:25.776868 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-dccfb33c-35e4-45a0-98ea-690bc1963013 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:33:25.787571 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:33:25.888066 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dccfb33c-35e4-45a0-98ea-690bc1963013 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:33:25.889507 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dccfb33c-35e4-45a0-98ea-690bc1963013 service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:33:25.907382 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 749/1493] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:33:25 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:33:35.550113 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=82 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:33:35.905117 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=50 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:35:03.843078 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37574510-5377-49cc-a429-dff37d939e82 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:35:03.844040 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37574510-5377-49cc-a429-dff37d939e82 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:35:03.844040 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37574510-5377-49cc-a429-dff37d939e82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:35:03.939483 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-37574510-5377-49cc-a429-dff37d939e82 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:35:03.940726 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-37574510-5377-49cc-a429-dff37d939e82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:35:04.065717 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 745/1494] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:35:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:35:04.074984 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2928cce-53b7-430a-910a-dd325371c0b2 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:35:04.075659 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2928cce-53b7-430a-910a-dd325371c0b2 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:35:04.075659 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2928cce-53b7-430a-910a-dd325371c0b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:35:04.194902 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-d2928cce-53b7-430a-910a-dd325371c0b2 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:35:04.198167 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-d2928cce-53b7-430a-910a-dd325371c0b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:35:04.339058 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 750/1495] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:35:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 265 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:35:04.344177 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37574510-5377-49cc-a429-dff37d939e82 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:35:04.345639 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-37574510-5377-49cc-a429-dff37d939e82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:35:04.359187 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37574510-5377-49cc-a429-dff37d939e82 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:35:04.466683 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:35:04.538171 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b88cdcb-63ce-4435-aed5-2b1c5f4bbfcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:35:04.539931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b88cdcb-63ce-4435-aed5-2b1c5f4bbfcc admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:35:04.540397 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b88cdcb-63ce-4435-aed5-2b1c5f4bbfcc admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:35:04.540957 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b88cdcb-63ce-4435-aed5-2b1c5f4bbfcc admin admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:35:04.542353 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b88cdcb-63ce-4435-aed5-2b1c5f4bbfcc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:35:04.542946 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b88cdcb-63ce-4435-aed5-2b1c5f4bbfcc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:35:04.545151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: WARNING py.warnings [None req-9b88cdcb-63ce-4435-aed5-2b1c5f4bbfcc admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jul 12 17:35:04.545151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: warnings.warn(msg) Jul 12 17:35:04.545151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: Jul 12 17:35:04.546952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b88cdcb-63ce-4435-aed5-2b1c5f4bbfcc admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:35:04.567515 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 746/1496] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:35:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:35:04.577095 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2928cce-53b7-430a-910a-dd325371c0b2 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:35:04.589657 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2928cce-53b7-430a-910a-dd325371c0b2 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:35:04.689931 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:35:04.736655 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f4a1fa-27ae-4ffb-9aef-8697888d9ea4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:35:04.737282 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f4a1fa-27ae-4ffb-9aef-8697888d9ea4 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:35:04.737575 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f4a1fa-27ae-4ffb-9aef-8697888d9ea4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:35:04.737857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f4a1fa-27ae-4ffb-9aef-8697888d9ea4 admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:35:04.738619 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f4a1fa-27ae-4ffb-9aef-8697888d9ea4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:35:04.740544 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: WARNING py.warnings [None req-33f4a1fa-27ae-4ffb-9aef-8697888d9ea4 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 12 17:35:04.740544 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: warnings.warn(msg) Jul 12 17:35:04.740544 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: Jul 12 17:35:04.741859 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f4a1fa-27ae-4ffb-9aef-8697888d9ea4 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:35:04.880147 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 751/1497] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:35:04 2022] POST /identity/v3/projects => generated 377 bytes in 304 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:35:04.885749 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b88cdcb-63ce-4435-aed5-2b1c5f4bbfcc admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:35:04.887851 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-9b88cdcb-63ce-4435-aed5-2b1c5f4bbfcc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:35:04.929265 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b88cdcb-63ce-4435-aed5-2b1c5f4bbfcc admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:35:05.056509 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc05179-1fa0-4a8e-bc9d-826626bee026 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:35:05.057299 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc05179-1fa0-4a8e-bc9d-826626bee026 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:35:05.057770 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc05179-1fa0-4a8e-bc9d-826626bee026 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:35:05.058120 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc05179-1fa0-4a8e-bc9d-826626bee026 admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:35:05.058957 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc05179-1fa0-4a8e-bc9d-826626bee026 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:35:05.060073 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: WARNING py.warnings [None req-8cc05179-1fa0-4a8e-bc9d-826626bee026 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 12 17:35:05.060073 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: warnings.warn(msg) Jul 12 17:35:05.060073 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: Jul 12 17:35:05.061520 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc05179-1fa0-4a8e-bc9d-826626bee026 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:35:05.205713 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 747/1498] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:35:04 2022] POST /identity/v3/users => generated 289 bytes in 321 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:35:05.210645 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f4a1fa-27ae-4ffb-9aef-8697888d9ea4 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:35:05.212693 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-33f4a1fa-27ae-4ffb-9aef-8697888d9ea4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:35:05.289293 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f4a1fa-27ae-4ffb-9aef-8697888d9ea4 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:35:05.393198 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:35:05.408390 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8c3ede-59f1-4306-8f47-1f4cb492415d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:35:05.409927 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8c3ede-59f1-4306-8f47-1f4cb492415d admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:35:05.410102 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8c3ede-59f1-4306-8f47-1f4cb492415d admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:35:05.410659 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8c3ede-59f1-4306-8f47-1f4cb492415d admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:35:05.412153 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d8c3ede-59f1-4306-8f47-1f4cb492415d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:35:05.413765 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: WARNING py.warnings [None req-9d8c3ede-59f1-4306-8f47-1f4cb492415d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jul 12 17:35:05.413765 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: warnings.warn(msg) Jul 12 17:35:05.413765 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: Jul 12 17:35:05.415969 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d8c3ede-59f1-4306-8f47-1f4cb492415d admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:35:05.442334 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 752/1499] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:35:05 2022] GET /identity/v3/roles => generated 1420 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 12 17:35:05.448559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc05179-1fa0-4a8e-bc9d-826626bee026 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:35:05.449109 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-8cc05179-1fa0-4a8e-bc9d-826626bee026 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:35:05.463893 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc05179-1fa0-4a8e-bc9d-826626bee026 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:35:05.494633 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:35:05.583221 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c469fcd1-5221-429d-bb9f-342a814966bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:35:05.584180 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c469fcd1-5221-429d-bb9f-342a814966bd admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:35:05.584464 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c469fcd1-5221-429d-bb9f-342a814966bd admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:35:05.584743 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c469fcd1-5221-429d-bb9f-342a814966bd admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:35:05.585494 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c469fcd1-5221-429d-bb9f-342a814966bd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:35:05.587272 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: WARNING py.warnings [None req-c469fcd1-5221-429d-bb9f-342a814966bd admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jul 12 17:35:05.587272 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: warnings.warn(msg) Jul 12 17:35:05.587272 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: Jul 12 17:35:05.588192 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c469fcd1-5221-429d-bb9f-342a814966bd admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:35:05.603211 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 748/1500] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:35:05 2022] GET /identity/v3/roles => generated 1420 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:35:05.609023 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8c3ede-59f1-4306-8f47-1f4cb492415d admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:35:05.610631 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-9d8c3ede-59f1-4306-8f47-1f4cb492415d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:35:05.657605 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8c3ede-59f1-4306-8f47-1f4cb492415d admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:35:05.774565 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6952e4-86c1-494b-b8ea-3badf6ba450d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:35:05.775507 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6952e4-86c1-494b-b8ea-3badf6ba450d admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:35:05.775793 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6952e4-86c1-494b-b8ea-3badf6ba450d admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:35:05.776074 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6952e4-86c1-494b-b8ea-3badf6ba450d admin admin] PATH_INFO: `/v3/projects/6d7d03c6d55a4d2c9f62ccdbd2301737/users/914b8caf34af4215acf0a336e449e6d6/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:35:05.803021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b6952e4-86c1-494b-b8ea-3badf6ba450d admin admin] RBAC: Authorizing `identity:create_grant(project_id=6d7d03c6d55a4d2c9f62ccdbd2301737, user_id=914b8caf34af4215acf0a336e449e6d6, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:35:05.804090 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: WARNING py.warnings [None req-0b6952e4-86c1-494b-b8ea-3badf6ba450d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 12 17:35:05.804090 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: warnings.warn(msg) Jul 12 17:35:05.804090 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: Jul 12 17:35:05.806271 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b6952e4-86c1-494b-b8ea-3badf6ba450d admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:35:05.912480 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 753/1501] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:35:05 2022] PUT /identity/v3/projects/6d7d03c6d55a4d2c9f62ccdbd2301737/users/914b8caf34af4215acf0a336e449e6d6/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 304 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:35:05.926569 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901cea66-4a94-4947-be83-c5c72c42a091 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:35:05.927220 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901cea66-4a94-4947-be83-c5c72c42a091 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:35:05.927569 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901cea66-4a94-4947-be83-c5c72c42a091 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:35:06.037210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-901cea66-4a94-4947-be83-c5c72c42a091 None None] MFA Rules not processed for user `914b8caf34af4215acf0a336e449e6d6`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:35:06.039251 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-901cea66-4a94-4947-be83-c5c72c42a091 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:35:06.191494 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 749/1502] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:35:05 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 267 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:35:06.207606 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50d9fe9-0d4e-4f55-83de-1b2366f7ee51 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:35:06.208117 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50d9fe9-0d4e-4f55-83de-1b2366f7ee51 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:35:06.208870 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50d9fe9-0d4e-4f55-83de-1b2366f7ee51 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:35:06.211614 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 754/1503] 10.222.0.16 () {54 vars in 963 bytes} [Tue Jul 12 17:35:06 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:35:06.217356 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022b718f-df59-4564-84dc-6d47505f853a None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:35:06.217651 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022b718f-df59-4564-84dc-6d47505f853a None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:35:06.217943 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022b718f-df59-4564-84dc-6d47505f853a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:35:06.254322 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:35:06.313376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-022b718f-df59-4564-84dc-6d47505f853a None None] MFA Rules not processed for user `f6a82b49464846d1a207189f17fd4cf7`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:35:06.314759 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-022b718f-df59-4564-84dc-6d47505f853a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:35:06.426774 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 750/1504] 10.222.0.16 () {58 vars in 1060 bytes} [Tue Jul 12 17:35:06 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:35:06.435699 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50d9fe9-0d4e-4f55-83de-1b2366f7ee51 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:35:06.437252 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-a50d9fe9-0d4e-4f55-83de-1b2366f7ee51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:35:06.497381 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:35:06.567461 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50d9fe9-0d4e-4f55-83de-1b2366f7ee51 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:35:06.659130 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ada123-61b0-4315-ab91-5274b65bba09 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:35:06.659794 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ada123-61b0-4315-ab91-5274b65bba09 service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:35:06.660085 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ada123-61b0-4315-ab91-5274b65bba09 service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:35:06.660364 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ada123-61b0-4315-ab91-5274b65bba09 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:35:06.662126 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-95ada123-61b0-4315-ab91-5274b65bba09 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:35:06.788969 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ada123-61b0-4315-ab91-5274b65bba09 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:35:06.791569 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ada123-61b0-4315-ab91-5274b65bba09 service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:35:06.795805 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:35:06.813443 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 755/1505] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:35:06 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:35:06.868953 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dcf2101-8e3e-45ad-9e4e-70e752b42347 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:35:06.869470 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dcf2101-8e3e-45ad-9e4e-70e752b42347 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:35:06.869964 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dcf2101-8e3e-45ad-9e4e-70e752b42347 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:35:06.872488 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 751/1506] 10.222.0.16 () {52 vars in 829 bytes} [Tue Jul 12 17:35:06 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:35:06.877256 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ada123-61b0-4315-ab91-5274b65bba09 service cinder] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:35:06.878662 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-95ada123-61b0-4315-ab91-5274b65bba09 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:35:07.134014 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ada123-61b0-4315-ab91-5274b65bba09 service cinder] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:35:07.263162 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb29c86-5a0e-4bea-8a42-f0fe59403586 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:35:07.264089 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb29c86-5a0e-4bea-8a42-f0fe59403586 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:35:07.264327 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb29c86-5a0e-4bea-8a42-f0fe59403586 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:35:07.264627 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb29c86-5a0e-4bea-8a42-f0fe59403586 admin admin] PATH_INFO: `/v3/users/914b8caf34af4215acf0a336e449e6d6` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:35:07.281160 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb29c86-5a0e-4bea-8a42-f0fe59403586 admin admin] RBAC: Authorizing `identity:delete_user(user_id=914b8caf34af4215acf0a336e449e6d6)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:35:07.282228 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: WARNING py.warnings [None req-bbb29c86-5a0e-4bea-8a42-f0fe59403586 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 12 17:35:07.282228 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: warnings.warn(msg) Jul 12 17:35:07.282228 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: Jul 12 17:35:07.283681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb29c86-5a0e-4bea-8a42-f0fe59403586 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:35:07.300962 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:35:07.593572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-bbb29c86-5a0e-4bea-8a42-f0fe59403586 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '914b8caf34af4215acf0a336e449e6d6'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:35:07.594106 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-bbb29c86-5a0e-4bea-8a42-f0fe59403586 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '914b8caf34af4215acf0a336e449e6d6'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:35:07.594925 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-bbb29c86-5a0e-4bea-8a42-f0fe59403586 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '914b8caf34af4215acf0a336e449e6d6'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:35:07.612105 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-bbb29c86-5a0e-4bea-8a42-f0fe59403586 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '914b8caf34af4215acf0a336e449e6d6'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:35:07.635538 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-bbb29c86-5a0e-4bea-8a42-f0fe59403586 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '914b8caf34af4215acf0a336e449e6d6'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:35:07.733768 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 756/1507] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:35:06 2022] DELETE /identity/v3/users/914b8caf34af4215acf0a336e449e6d6 => generated 0 bytes in 857 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:35:07.738570 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eafb3e6-c92a-4fc0-8d5f-caebc2005725 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:35:07.741742 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eafb3e6-c92a-4fc0-8d5f-caebc2005725 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:35:07.741742 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eafb3e6-c92a-4fc0-8d5f-caebc2005725 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:35:07.812317 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-4eafb3e6-c92a-4fc0-8d5f-caebc2005725 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:35:07.813603 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-4eafb3e6-c92a-4fc0-8d5f-caebc2005725 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:35:07.918029 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:35:07.937137 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 752/1508] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:35:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:35:07.948483 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb29c86-5a0e-4bea-8a42-f0fe59403586 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:35:07.949732 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-bbb29c86-5a0e-4bea-8a42-f0fe59403586 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:35:08.078582 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb29c86-5a0e-4bea-8a42-f0fe59403586 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:35:08.149344 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:35:08.171378 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b83c86-4235-463e-8059-83dfbbda91ec service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:35:08.171941 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b83c86-4235-463e-8059-83dfbbda91ec service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:35:08.172224 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b83c86-4235-463e-8059-83dfbbda91ec service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:35:08.172502 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b83c86-4235-463e-8059-83dfbbda91ec service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:35:08.174290 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-15b83c86-4235-463e-8059-83dfbbda91ec service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:35:08.314742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b83c86-4235-463e-8059-83dfbbda91ec service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:35:08.316324 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b83c86-4235-463e-8059-83dfbbda91ec service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:35:08.341621 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 757/1509] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:35:07 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 394 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:35:09.092452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eafb3e6-c92a-4fc0-8d5f-caebc2005725 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:35:09.094305 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-4eafb3e6-c92a-4fc0-8d5f-caebc2005725 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:35:09.117580 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eafb3e6-c92a-4fc0-8d5f-caebc2005725 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:35:09.260602 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c414a03-f001-4d5c-8018-c89bfabd2f88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:35:09.262044 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c414a03-f001-4d5c-8018-c89bfabd2f88 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:35:09.262553 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c414a03-f001-4d5c-8018-c89bfabd2f88 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:35:09.263044 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c414a03-f001-4d5c-8018-c89bfabd2f88 admin admin] PATH_INFO: `/v3/projects/6d7d03c6d55a4d2c9f62ccdbd2301737` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:35:09.282271 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c414a03-f001-4d5c-8018-c89bfabd2f88 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6d7d03c6d55a4d2c9f62ccdbd2301737)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:35:09.283886 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: WARNING py.warnings [None req-9c414a03-f001-4d5c-8018-c89bfabd2f88 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 12 17:35:09.283886 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: warnings.warn(msg) Jul 12 17:35:09.283886 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: Jul 12 17:35:09.286028 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c414a03-f001-4d5c-8018-c89bfabd2f88 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:35:09.451544 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-9c414a03-f001-4d5c-8018-c89bfabd2f88 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6d7d03c6d55a4d2c9f62ccdbd2301737', 'request_id': 'req-9c414a03-f001-4d5c-8018-c89bfabd2f88'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:35:09.482232 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-9c414a03-f001-4d5c-8018-c89bfabd2f88 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6d7d03c6d55a4d2c9f62ccdbd2301737', 'request_id': 'req-9c414a03-f001-4d5c-8018-c89bfabd2f88'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:35:09.494280 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-9c414a03-f001-4d5c-8018-c89bfabd2f88 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6d7d03c6d55a4d2c9f62ccdbd2301737', 'request_id': 'req-9c414a03-f001-4d5c-8018-c89bfabd2f88'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:35:09.621118 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-9c414a03-f001-4d5c-8018-c89bfabd2f88 admin admin] The token cache is being invalidate because project 6d7d03c6d55a4d2c9f62ccdbd2301737 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:35:09.621483 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-9c414a03-f001-4d5c-8018-c89bfabd2f88 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:35:09.621963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-9c414a03-f001-4d5c-8018-c89bfabd2f88 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:35:09.623730 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 753/1510] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:35:09 2022] DELETE /identity/v3/projects/6d7d03c6d55a4d2c9f62ccdbd2301737 => generated 0 bytes in 532 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:35:18.338740 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=62 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:35:19.614943 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:36:55.732860 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeac7901-7198-4805-ba81-c5a83125cc00 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:36:55.733503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeac7901-7198-4805-ba81-c5a83125cc00 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:36:55.733503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeac7901-7198-4805-ba81-c5a83125cc00 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:36:55.734834 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 758/1511] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 12 17:36:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 12 17:36:55.740400 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac0b791-bd49-4ef7-9de1-a88239e0c95e None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:36:55.740883 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac0b791-bd49-4ef7-9de1-a88239e0c95e None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:36:55.740987 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac0b791-bd49-4ef7-9de1-a88239e0c95e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:36:55.837845 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-8ac0b791-bd49-4ef7-9de1-a88239e0c95e None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:36:55.839013 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-8ac0b791-bd49-4ef7-9de1-a88239e0c95e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:36:55.973805 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 754/1512] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 12 17:36:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:36:56.132708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeac7901-7198-4805-ba81-c5a83125cc00 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:36:56.134906 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-eeac7901-7198-4805-ba81-c5a83125cc00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:36:56.255968 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeac7901-7198-4805-ba81-c5a83125cc00 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:36:56.261861 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d74bfe1b-e8de-466c-a261-ea5d47eba5d5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:36:56.262561 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74bfe1b-e8de-466c-a261-ea5d47eba5d5 service nova] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:36:56.262855 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74bfe1b-e8de-466c-a261-ea5d47eba5d5 service nova] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:36:56.263139 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d74bfe1b-e8de-466c-a261-ea5d47eba5d5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:36:56.272631 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d74bfe1b-e8de-466c-a261-ea5d47eba5d5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:36:56.274384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d74bfe1b-e8de-466c-a261-ea5d47eba5d5 service nova] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:36:56.285094 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 759/1513] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:36:56 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:36:57.175178 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063c9480-b685-4b21-ab8a-0ec7d9ea21ef None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:36:57.175785 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063c9480-b685-4b21-ab8a-0ec7d9ea21ef None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:36:57.175894 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8c5ab5-7a5e-4239-b36e-5c1e7fd8f294 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:36:57.176307 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063c9480-b685-4b21-ab8a-0ec7d9ea21ef None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:36:57.176413 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8c5ab5-7a5e-4239-b36e-5c1e7fd8f294 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:36:57.176413 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8c5ab5-7a5e-4239-b36e-5c1e7fd8f294 None None] PATH_INFO: `/v3` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:36:57.177269 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 755/1514] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 12 17:36:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:36:57.177388 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 760/1515] 10.222.0.36 () {54 vars in 925 bytes} [Tue Jul 12 17:36:57 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:36:57.182012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0f2f99-292d-4ff4-a5ab-44c7bcafe45c None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:36:57.182305 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0f2f99-292d-4ff4-a5ab-44c7bcafe45c None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:36:57.182682 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0f2f99-292d-4ff4-a5ab-44c7bcafe45c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:36:57.183876 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b1c628-f0b0-4a06-9ee8-8cd03f2e5c6f None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:36:57.184236 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b1c628-f0b0-4a06-9ee8-8cd03f2e5c6f None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:36:57.184592 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b1c628-f0b0-4a06-9ee8-8cd03f2e5c6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:36:57.202671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-df0f2f99-292d-4ff4-a5ab-44c7bcafe45c None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:36:57.203930 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-df0f2f99-292d-4ff4-a5ab-44c7bcafe45c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:36:57.214962 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 756/1516] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 12 17:36:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:36:57.251985 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-86b1c628-f0b0-4a06-9ee8-8cd03f2e5c6f None None] MFA Rules not processed for user `f623be28c83247c78d3ea6873c347fe0`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:36:57.253946 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-86b1c628-f0b0-4a06-9ee8-8cd03f2e5c6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:36:57.331922 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:36:57.337906 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 761/1517] 10.222.0.36 () {58 vars in 1015 bytes} [Tue Jul 12 17:36:57 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:36:57.373599 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0f2f99-292d-4ff4-a5ab-44c7bcafe45c None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:36:57.386958 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0f2f99-292d-4ff4-a5ab-44c7bcafe45c None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:36:57.392601 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fc995c6-d699-431f-9061-d041d066a18d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:36:57.393928 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc995c6-d699-431f-9061-d041d066a18d service nova] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:36:57.394217 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc995c6-d699-431f-9061-d041d066a18d service nova] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:36:57.394534 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc995c6-d699-431f-9061-d041d066a18d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:36:57.403635 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fc995c6-d699-431f-9061-d041d066a18d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:36:57.405238 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fc995c6-d699-431f-9061-d041d066a18d service nova] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:36:57.415996 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 757/1518] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:36:57 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:37:06.654167 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2aa91f2-9156-4ddb-b149-9bb191183bf1 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:37:06.654814 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2aa91f2-9156-4ddb-b149-9bb191183bf1 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:37:06.655048 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2aa91f2-9156-4ddb-b149-9bb191183bf1 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:37:06.656996 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 762/1519] 10.222.0.16 () {54 vars in 960 bytes} [Tue Jul 12 17:37:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 12 17:37:06.664037 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce09d41-e150-4a7f-b325-cfb51108f64b None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:37:06.664647 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce09d41-e150-4a7f-b325-cfb51108f64b None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:37:06.664647 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce09d41-e150-4a7f-b325-cfb51108f64b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:37:06.685004 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-6ce09d41-e150-4a7f-b325-cfb51108f64b None None] MFA Rules not processed for user `f623be28c83247c78d3ea6873c347fe0`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:37:06.686233 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-6ce09d41-e150-4a7f-b325-cfb51108f64b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:37:06.696514 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 758/1520] 10.222.0.16 () {58 vars in 1057 bytes} [Tue Jul 12 17:37:06 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:37:06.707873 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2aa91f2-9156-4ddb-b149-9bb191183bf1 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:37:06.731271 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2aa91f2-9156-4ddb-b149-9bb191183bf1 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:37:06.738817 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e402fe-298b-434d-abad-22cd8f7a52e7 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f623be28c83247c78d3ea6873c347fe0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:37:06.739541 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e402fe-298b-434d-abad-22cd8f7a52e7 service placement] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:37:06.739963 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e402fe-298b-434d-abad-22cd8f7a52e7 service placement] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:37:06.740333 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e402fe-298b-434d-abad-22cd8f7a52e7 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:37:06.751002 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e402fe-298b-434d-abad-22cd8f7a52e7 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:37:06.753009 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e402fe-298b-434d-abad-22cd8f7a52e7 service placement] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:37:06.764590 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 763/1521] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:37:06 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 58 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:37:08.771532 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dabe58-8fc9-40e2-9a68-2a0960266bee None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:37:08.772146 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dabe58-8fc9-40e2-9a68-2a0960266bee None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:37:08.772323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dabe58-8fc9-40e2-9a68-2a0960266bee None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:37:08.774130 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 759/1522] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 12 17:37:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 12 17:37:08.779901 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2f09da-75cc-4817-b25a-65ff19ae3c61 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:37:08.780534 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2f09da-75cc-4817-b25a-65ff19ae3c61 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:37:08.780534 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2f09da-75cc-4817-b25a-65ff19ae3c61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:37:08.801740 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-fa2f09da-75cc-4817-b25a-65ff19ae3c61 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:37:08.803048 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-fa2f09da-75cc-4817-b25a-65ff19ae3c61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:37:08.811500 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 764/1523] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 12 17:37:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:37:08.978736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9dabe58-8fc9-40e2-9a68-2a0960266bee None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:37:08.996803 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9dabe58-8fc9-40e2-9a68-2a0960266bee None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:37:09.003620 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8b5b101-a11f-469f-9e59-880cb0b00c2b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:37:09.004283 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b5b101-a11f-469f-9e59-880cb0b00c2b service nova] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:37:09.004576 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b5b101-a11f-469f-9e59-880cb0b00c2b service nova] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:37:09.004862 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b5b101-a11f-469f-9e59-880cb0b00c2b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:37:09.014796 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b5b101-a11f-469f-9e59-880cb0b00c2b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:37:09.016452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8b5b101-a11f-469f-9e59-880cb0b00c2b service nova] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:37:09.028366 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 760/1524] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:37:08 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:37:10.026216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ae15bf-2f7c-465f-8e9d-15b6235d1b0d None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:37:10.026920 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ae15bf-2f7c-465f-8e9d-15b6235d1b0d None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:37:10.026920 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ae15bf-2f7c-465f-8e9d-15b6235d1b0d None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:37:10.028371 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 765/1525] 127.0.0.1 () {54 vars in 901 bytes} [Tue Jul 12 17:37:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 12 17:37:10.035300 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ee3568-f613-4516-a0bb-031845c6c42c None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:37:10.035850 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ee3568-f613-4516-a0bb-031845c6c42c None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:37:10.036330 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ee3568-f613-4516-a0bb-031845c6c42c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:37:10.061595 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-12ee3568-f613-4516-a0bb-031845c6c42c None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:37:10.063046 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-12ee3568-f613-4516-a0bb-031845c6c42c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:37:10.072109 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 761/1526] 127.0.0.1 () {58 vars in 998 bytes} [Tue Jul 12 17:37:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:37:10.245937 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ae15bf-2f7c-465f-8e9d-15b6235d1b0d None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:37:10.264499 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ae15bf-2f7c-465f-8e9d-15b6235d1b0d None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:37:10.271091 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ad9157-2b4b-4309-8ff6-c030ca907f94 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:37:10.271757 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ad9157-2b4b-4309-8ff6-c030ca907f94 service nova] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:37:10.272043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ad9157-2b4b-4309-8ff6-c030ca907f94 service nova] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:37:10.272325 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ad9157-2b4b-4309-8ff6-c030ca907f94 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:37:10.281635 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83ad9157-2b4b-4309-8ff6-c030ca907f94 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:37:10.283856 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83ad9157-2b4b-4309-8ff6-c030ca907f94 service nova] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:37:10.294550 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 766/1527] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:37:10 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:37:14.471477 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e1ff0a-8f98-4fcc-8ac8-16ce3973e74a None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:37:14.472071 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e1ff0a-8f98-4fcc-8ac8-16ce3973e74a None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:37:14.472462 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e1ff0a-8f98-4fcc-8ac8-16ce3973e74a None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:37:14.475075 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 762/1528] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 12 17:37:14 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:37:14.481563 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1681d7ea-ddf2-4292-a442-e2100817017a None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:37:14.482199 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1681d7ea-ddf2-4292-a442-e2100817017a None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:37:14.482199 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1681d7ea-ddf2-4292-a442-e2100817017a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:37:14.503328 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-1681d7ea-ddf2-4292-a442-e2100817017a None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:37:14.504868 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-1681d7ea-ddf2-4292-a442-e2100817017a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:37:14.515313 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 767/1529] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 12 17:37:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:37:14.603983 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e1ff0a-8f98-4fcc-8ac8-16ce3973e74a None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:37:14.605763 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-d9e1ff0a-8f98-4fcc-8ac8-16ce3973e74a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:37:14.637666 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:37:14.707558 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e1ff0a-8f98-4fcc-8ac8-16ce3973e74a None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:37:14.711616 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d49577c-2888-4243-868e-939bdb4b06e6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:37:14.712467 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d49577c-2888-4243-868e-939bdb4b06e6 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:37:14.712794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d49577c-2888-4243-868e-939bdb4b06e6 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:37:14.713111 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d49577c-2888-4243-868e-939bdb4b06e6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:37:14.726257 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d49577c-2888-4243-868e-939bdb4b06e6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:37:14.728036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d49577c-2888-4243-868e-939bdb4b06e6 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:37:14.740565 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 763/1530] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:37:14 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:37:16.905147 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e885b093-c205-402e-9893-39018de433ee None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:37:16.905954 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e885b093-c205-402e-9893-39018de433ee None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:37:16.906052 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e885b093-c205-402e-9893-39018de433ee None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:37:16.909877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 768/1531] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 12 17:37:16 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 12 17:37:16.915422 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ecf206-6470-4bb1-b39c-245ee6f63054 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:37:16.915992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ecf206-6470-4bb1-b39c-245ee6f63054 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:37:16.916091 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ecf206-6470-4bb1-b39c-245ee6f63054 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:37:16.936140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-f9ecf206-6470-4bb1-b39c-245ee6f63054 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:37:16.937472 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-f9ecf206-6470-4bb1-b39c-245ee6f63054 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:37:16.945875 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 764/1532] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 12 17:37:16 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:37:17.000739 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e885b093-c205-402e-9893-39018de433ee None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:37:17.013931 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e885b093-c205-402e-9893-39018de433ee None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:37:17.018305 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef95477f-be3b-42f3-bd2b-a1adade08a84 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:37:17.019006 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef95477f-be3b-42f3-bd2b-a1adade08a84 service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:37:17.019298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef95477f-be3b-42f3-bd2b-a1adade08a84 service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:37:17.019582 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef95477f-be3b-42f3-bd2b-a1adade08a84 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:37:17.030179 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef95477f-be3b-42f3-bd2b-a1adade08a84 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:37:17.031699 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef95477f-be3b-42f3-bd2b-a1adade08a84 service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:37:17.043913 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 769/1533] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:37:16 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:37:20.591817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5109f225-85c7-47a0-bedf-be98278dac15 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:37:20.592378 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5109f225-85c7-47a0-bedf-be98278dac15 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:37:20.592378 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5109f225-85c7-47a0-bedf-be98278dac15 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:37:20.593990 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 765/1534] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 12 17:37:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:37:20.600749 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cb3a4a-236b-48fa-b9dc-01228789e7e4 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:37:20.601269 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cb3a4a-236b-48fa-b9dc-01228789e7e4 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:37:20.601404 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21cb3a4a-236b-48fa-b9dc-01228789e7e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:37:20.621160 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-21cb3a4a-236b-48fa-b9dc-01228789e7e4 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:37:20.622380 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-21cb3a4a-236b-48fa-b9dc-01228789e7e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:37:20.630808 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 770/1535] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 12 17:37:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:37:20.950049 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5109f225-85c7-47a0-bedf-be98278dac15 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:37:20.952103 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-5109f225-85c7-47a0-bedf-be98278dac15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:37:21.053628 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5109f225-85c7-47a0-bedf-be98278dac15 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:37:21.057573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a63fa8-f146-4eff-95d0-f5564dbac594 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:37:21.058288 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a63fa8-f146-4eff-95d0-f5564dbac594 service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:37:21.058617 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a63fa8-f146-4eff-95d0-f5564dbac594 service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:37:21.058902 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a63fa8-f146-4eff-95d0-f5564dbac594 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:37:21.069238 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a63fa8-f146-4eff-95d0-f5564dbac594 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:37:21.072081 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a63fa8-f146-4eff-95d0-f5564dbac594 service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:37:21.085705 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 766/1536] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:37:20 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:37:23.253357 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f438ba95-a03d-4287-970d-bd73c3b83357 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:37:23.253945 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f438ba95-a03d-4287-970d-bd73c3b83357 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:37:23.253945 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f438ba95-a03d-4287-970d-bd73c3b83357 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:37:23.255349 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 771/1537] 127.0.0.1 () {54 vars in 916 bytes} [Tue Jul 12 17:37:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:37:23.262071 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42957e23-58a8-4749-b836-f72987b97fbc None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:37:23.262674 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42957e23-58a8-4749-b836-f72987b97fbc None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:37:23.263171 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42957e23-58a8-4749-b836-f72987b97fbc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:37:23.294869 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-42957e23-58a8-4749-b836-f72987b97fbc None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:37:23.296265 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-42957e23-58a8-4749-b836-f72987b97fbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:37:23.306042 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 767/1538] 127.0.0.1 () {58 vars in 1013 bytes} [Tue Jul 12 17:37:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:37:23.620052 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f438ba95-a03d-4287-970d-bd73c3b83357 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:37:23.622015 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-f438ba95-a03d-4287-970d-bd73c3b83357 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:37:23.641390 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f438ba95-a03d-4287-970d-bd73c3b83357 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:37:23.647121 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-287adc7a-2fc9-4e31-80cf-c785f2945892 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:37:23.648198 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287adc7a-2fc9-4e31-80cf-c785f2945892 service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:37:23.648719 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287adc7a-2fc9-4e31-80cf-c785f2945892 service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:37:23.649221 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287adc7a-2fc9-4e31-80cf-c785f2945892 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:37:23.666699 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-287adc7a-2fc9-4e31-80cf-c785f2945892 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:37:23.669589 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-287adc7a-2fc9-4e31-80cf-c785f2945892 service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:37:23.688041 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 772/1539] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:37:23 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 68 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:37:33.287428 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=66 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:37:33.680242 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=34 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:38:54.933490 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9415758a-c1d7-41c0-9df1-f7ca80a02e0e None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:54.935235 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9415758a-c1d7-41c0-9df1-f7ca80a02e0e None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:54.936091 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9415758a-c1d7-41c0-9df1-f7ca80a02e0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:54.983274 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-9415758a-c1d7-41c0-9df1-f7ca80a02e0e None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:38:54.984528 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-9415758a-c1d7-41c0-9df1-f7ca80a02e0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:38:54.996166 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 768/1540] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:38:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 65 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:38:55.007327 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eca0ce9-782a-4b93-b7ee-a81c7dad5576 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:55.008891 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eca0ce9-782a-4b93-b7ee-a81c7dad5576 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:55.009682 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eca0ce9-782a-4b93-b7ee-a81c7dad5576 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:55.052986 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-7eca0ce9-782a-4b93-b7ee-a81c7dad5576 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:38:55.056143 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-7eca0ce9-782a-4b93-b7ee-a81c7dad5576 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:38:55.078016 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 773/1541] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:38:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 73 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:38:55.082496 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9415758a-c1d7-41c0-9df1-f7ca80a02e0e None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:55.094161 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9415758a-c1d7-41c0-9df1-f7ca80a02e0e None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:55.099214 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c19ecf-9061-4851-a6a0-5d7d0d8aff27 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:55.100195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c19ecf-9061-4851-a6a0-5d7d0d8aff27 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:55.101082 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c19ecf-9061-4851-a6a0-5d7d0d8aff27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:55.105044 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82cf69a8-5063-42f8-a49b-4d6d81c5f6a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:55.105044 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82cf69a8-5063-42f8-a49b-4d6d81c5f6a3 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:55.105458 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82cf69a8-5063-42f8-a49b-4d6d81c5f6a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:55.105458 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82cf69a8-5063-42f8-a49b-4d6d81c5f6a3 admin admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:55.106236 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82cf69a8-5063-42f8-a49b-4d6d81c5f6a3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:38:55.106881 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82cf69a8-5063-42f8-a49b-4d6d81c5f6a3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:55.109111 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82cf69a8-5063-42f8-a49b-4d6d81c5f6a3 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:55.136510 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 769/1542] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:38:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:38:55.136510 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc1b9f5d-3aff-48b5-a1c5-80d472cd2c65 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:55.136510 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc1b9f5d-3aff-48b5-a1c5-80d472cd2c65 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:55.137066 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc1b9f5d-3aff-48b5-a1c5-80d472cd2c65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:55.151488 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-20c19ecf-9061-4851-a6a0-5d7d0d8aff27 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:38:55.153995 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-20c19ecf-9061-4851-a6a0-5d7d0d8aff27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:38:55.168580 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-cc1b9f5d-3aff-48b5-a1c5-80d472cd2c65 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:38:55.169833 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-cc1b9f5d-3aff-48b5-a1c5-80d472cd2c65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:38:55.176727 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 774/1543] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:38:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:38:55.178827 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74dbff56-dbff-4905-b7dc-9e17acbd5ad9 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:55.179114 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74dbff56-dbff-4905-b7dc-9e17acbd5ad9 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:55.179432 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74dbff56-dbff-4905-b7dc-9e17acbd5ad9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:55.189504 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 770/1544] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:38:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:38:55.190879 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc1b9f5d-3aff-48b5-a1c5-80d472cd2c65 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:55.218489 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc1b9f5d-3aff-48b5-a1c5-80d472cd2c65 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:55.220316 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-74dbff56-dbff-4905-b7dc-9e17acbd5ad9 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:38:55.221668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-74dbff56-dbff-4905-b7dc-9e17acbd5ad9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:38:55.230349 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76df642-5786-4269-a160-0fa20f3c9394 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:55.231072 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76df642-5786-4269-a160-0fa20f3c9394 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:55.231358 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76df642-5786-4269-a160-0fa20f3c9394 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:55.231639 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76df642-5786-4269-a160-0fa20f3c9394 admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:55.232351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b76df642-5786-4269-a160-0fa20f3c9394 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:55.234062 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: WARNING py.warnings [None req-b76df642-5786-4269-a160-0fa20f3c9394 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 12 17:38:55.234062 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: warnings.warn(msg) Jul 12 17:38:55.234062 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: Jul 12 17:38:55.237138 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b76df642-5786-4269-a160-0fa20f3c9394 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:55.245642 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 775/1545] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:38:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 68 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:38:55.252730 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57e9139-fd35-4ed7-b843-74354760df51 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:55.253115 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57e9139-fd35-4ed7-b843-74354760df51 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:55.253341 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57e9139-fd35-4ed7-b843-74354760df51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:55.296660 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-c57e9139-fd35-4ed7-b843-74354760df51 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:38:55.297877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-c57e9139-fd35-4ed7-b843-74354760df51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:38:55.318124 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 776/1546] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:38:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 72 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:38:55.319127 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f77d44-c623-4092-b732-13b3a15e6ded None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:55.319479 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f77d44-c623-4092-b732-13b3a15e6ded None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:55.319758 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f77d44-c623-4092-b732-13b3a15e6ded None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:55.346385 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-32f77d44-c623-4092-b732-13b3a15e6ded None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:38:55.347939 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-32f77d44-c623-4092-b732-13b3a15e6ded None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:38:55.364298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 777/1547] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:38:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:38:55.365628 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a26bd6-557b-4527-af8a-5c437ecf87cc None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:55.365918 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a26bd6-557b-4527-af8a-5c437ecf87cc None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:55.366188 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a26bd6-557b-4527-af8a-5c437ecf87cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:55.392951 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-c1a26bd6-557b-4527-af8a-5c437ecf87cc None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:38:55.394165 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-c1a26bd6-557b-4527-af8a-5c437ecf87cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:38:55.411568 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 771/1548] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:38:55 2022] POST /identity/v3/projects => generated 391 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:38:55.412802 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76df642-5786-4269-a160-0fa20f3c9394 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:55.432078 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 778/1549] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:38:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 66 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:38:55.433261 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a26bd6-557b-4527-af8a-5c437ecf87cc None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:55.465431 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a26bd6-557b-4527-af8a-5c437ecf87cc None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:55.609161 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76df642-5786-4269-a160-0fa20f3c9394 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:55.631191 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8f81e7-b5eb-4791-9ccb-67fb9c2832a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:55.633242 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8f81e7-b5eb-4791-9ccb-67fb9c2832a7 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:55.634223 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8f81e7-b5eb-4791-9ccb-67fb9c2832a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:55.639401 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8f81e7-b5eb-4791-9ccb-67fb9c2832a7 admin admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:55.641921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce8f81e7-b5eb-4791-9ccb-67fb9c2832a7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:38:55.643828 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce8f81e7-b5eb-4791-9ccb-67fb9c2832a7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:55.649920 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce8f81e7-b5eb-4791-9ccb-67fb9c2832a7 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:55.665842 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3180c234-9732-450e-8dd9-ecbfb1fa21eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:55.667218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3180c234-9732-450e-8dd9-ecbfb1fa21eb admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:55.667677 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3180c234-9732-450e-8dd9-ecbfb1fa21eb admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:55.668118 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3180c234-9732-450e-8dd9-ecbfb1fa21eb admin admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:55.669014 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3180c234-9732-450e-8dd9-ecbfb1fa21eb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:38:55.671071 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3180c234-9732-450e-8dd9-ecbfb1fa21eb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:55.672288 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: WARNING py.warnings [None req-3180c234-9732-450e-8dd9-ecbfb1fa21eb admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jul 12 17:38:55.672288 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: warnings.warn(msg) Jul 12 17:38:55.672288 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: Jul 12 17:38:55.675547 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3180c234-9732-450e-8dd9-ecbfb1fa21eb admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:55.679764 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 772/1550] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:38:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:38:55.681021 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8f81e7-b5eb-4791-9ccb-67fb9c2832a7 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:55.700336 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 779/1551] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:38:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:38:55.702801 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8f81e7-b5eb-4791-9ccb-67fb9c2832a7 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:55.703821 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3180c234-9732-450e-8dd9-ecbfb1fa21eb admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:55.715888 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785ad053-f07b-4cf5-864e-e515372ca726 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:55.717198 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785ad053-f07b-4cf5-864e-e515372ca726 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:55.718033 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785ad053-f07b-4cf5-864e-e515372ca726 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:55.718498 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785ad053-f07b-4cf5-864e-e515372ca726 admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:55.719365 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-785ad053-f07b-4cf5-864e-e515372ca726 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:55.722365 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-785ad053-f07b-4cf5-864e-e515372ca726 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:55.726141 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3180c234-9732-450e-8dd9-ecbfb1fa21eb admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:55.731285 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c6e4a85-f162-40f0-9c00-b63ddf1d129f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:55.732010 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6e4a85-f162-40f0-9c00-b63ddf1d129f admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:55.732319 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6e4a85-f162-40f0-9c00-b63ddf1d129f admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:55.732552 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c6e4a85-f162-40f0-9c00-b63ddf1d129f admin admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:55.733280 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c6e4a85-f162-40f0-9c00-b63ddf1d129f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:38:55.733612 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c6e4a85-f162-40f0-9c00-b63ddf1d129f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:55.735207 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c6e4a85-f162-40f0-9c00-b63ddf1d129f admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:55.749988 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 780/1552] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:38:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:38:55.758079 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb54adbf-ce48-4147-b823-038d26b616b9 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:55.759403 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb54adbf-ce48-4147-b823-038d26b616b9 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:55.759403 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb54adbf-ce48-4147-b823-038d26b616b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:55.777572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-cb54adbf-ce48-4147-b823-038d26b616b9 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:38:55.778849 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-cb54adbf-ce48-4147-b823-038d26b616b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:38:55.790575 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 781/1553] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:38:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:38:55.790575 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb54adbf-ce48-4147-b823-038d26b616b9 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:55.803026 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb54adbf-ce48-4147-b823-038d26b616b9 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:55.808390 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc095969-e836-4844-be02-df2cb7e7f80f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:55.809255 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc095969-e836-4844-be02-df2cb7e7f80f admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:55.809798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc095969-e836-4844-be02-df2cb7e7f80f admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:55.810080 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc095969-e836-4844-be02-df2cb7e7f80f admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:55.810823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc095969-e836-4844-be02-df2cb7e7f80f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:55.816754 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc095969-e836-4844-be02-df2cb7e7f80f admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:55.850013 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 773/1554] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:38:55 2022] POST /identity/v3/users => generated 296 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:38:55.850866 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785ad053-f07b-4cf5-864e-e515372ca726 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:55.860903 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785ad053-f07b-4cf5-864e-e515372ca726 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:55.866286 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af05f59-d4a0-460d-b8ff-64f051ead373 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:55.867021 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af05f59-d4a0-460d-b8ff-64f051ead373 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:55.867336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af05f59-d4a0-460d-b8ff-64f051ead373 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:55.867624 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1af05f59-d4a0-460d-b8ff-64f051ead373 admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:55.868335 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af05f59-d4a0-460d-b8ff-64f051ead373 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:55.870259 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1af05f59-d4a0-460d-b8ff-64f051ead373 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:55.948269 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 782/1555] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:38:55 2022] POST /identity/v3/projects => generated 379 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:38:55.949785 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058e4afe-931a-46e2-8abb-c148d74cdc95 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:55.950077 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058e4afe-931a-46e2-8abb-c148d74cdc95 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:55.950361 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058e4afe-931a-46e2-8abb-c148d74cdc95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:55.969205 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-058e4afe-931a-46e2-8abb-c148d74cdc95 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:38:55.970336 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-058e4afe-931a-46e2-8abb-c148d74cdc95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:38:55.976915 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 774/1556] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:38:55 2022] POST /identity/v3/projects => generated 373 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:38:55.977936 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af05f59-d4a0-460d-b8ff-64f051ead373 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:56.037983 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:38:56.042344 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1af05f59-d4a0-460d-b8ff-64f051ead373 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:56.046784 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 783/1557] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:38:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:38:56.047292 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4cecbf1-56ee-44aa-ab10-79d23c17ef5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:56.047806 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4cecbf1-56ee-44aa-ab10-79d23c17ef5a admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:56.048107 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4cecbf1-56ee-44aa-ab10-79d23c17ef5a admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:56.048296 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605424b3-177e-473f-8fa9-dc506f7a7577 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:56.048403 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4cecbf1-56ee-44aa-ab10-79d23c17ef5a admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:56.048593 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605424b3-177e-473f-8fa9-dc506f7a7577 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:56.048882 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605424b3-177e-473f-8fa9-dc506f7a7577 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:56.049193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4cecbf1-56ee-44aa-ab10-79d23c17ef5a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:56.051199 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4cecbf1-56ee-44aa-ab10-79d23c17ef5a admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:56.069520 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-605424b3-177e-473f-8fa9-dc506f7a7577 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:38:56.071069 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-605424b3-177e-473f-8fa9-dc506f7a7577 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:38:56.079427 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 784/1558] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:38:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:38:56.081485 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91bd047-2f80-4155-b0be-873a3e6d9a97 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:56.081831 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91bd047-2f80-4155-b0be-873a3e6d9a97 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:56.082113 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a91bd047-2f80-4155-b0be-873a3e6d9a97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:56.095027 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:38:56.101970 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-a91bd047-2f80-4155-b0be-873a3e6d9a97 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:38:56.103222 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-a91bd047-2f80-4155-b0be-873a3e6d9a97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:38:56.111729 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 785/1559] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:38:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:38:56.113163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f08cba1-4221-45ac-b1f0-fae92a5a6ed3 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:56.113478 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f08cba1-4221-45ac-b1f0-fae92a5a6ed3 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:56.113758 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f08cba1-4221-45ac-b1f0-fae92a5a6ed3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:56.131885 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-0f08cba1-4221-45ac-b1f0-fae92a5a6ed3 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:38:56.133134 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-0f08cba1-4221-45ac-b1f0-fae92a5a6ed3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:38:56.141157 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 786/1560] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:38:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:38:56.141999 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f08cba1-4221-45ac-b1f0-fae92a5a6ed3 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:56.148040 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 775/1561] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:38:55 2022] POST /identity/v3/projects => generated 383 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:38:56.148997 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4cecbf1-56ee-44aa-ab10-79d23c17ef5a admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:56.153996 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f08cba1-4221-45ac-b1f0-fae92a5a6ed3 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:56.215525 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc05a78-38d1-4392-b558-302f0a40134f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:56.216805 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc05a78-38d1-4392-b558-302f0a40134f admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:56.217096 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc05a78-38d1-4392-b558-302f0a40134f admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:56.217379 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc05a78-38d1-4392-b558-302f0a40134f admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:56.218229 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc05a78-38d1-4392-b558-302f0a40134f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:56.219835 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4cecbf1-56ee-44aa-ab10-79d23c17ef5a admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:56.219952 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc05a78-38d1-4392-b558-302f0a40134f admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:56.225617 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f876fde-124e-44d0-9f13-badf9fd96e44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:56.226225 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f876fde-124e-44d0-9f13-badf9fd96e44 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:56.226551 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f876fde-124e-44d0-9f13-badf9fd96e44 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:56.226831 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f876fde-124e-44d0-9f13-badf9fd96e44 admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:56.227547 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f876fde-124e-44d0-9f13-badf9fd96e44 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:56.229524 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f876fde-124e-44d0-9f13-badf9fd96e44 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:56.231430 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 787/1562] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:38:56 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:38:56.232505 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc05a78-38d1-4392-b558-302f0a40134f admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:56.243892 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc05a78-38d1-4392-b558-302f0a40134f admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:56.248879 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95be025a-ff27-4f65-9202-4c6ca0f079bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:56.249575 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95be025a-ff27-4f65-9202-4c6ca0f079bb admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:56.249866 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95be025a-ff27-4f65-9202-4c6ca0f079bb admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:56.250148 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95be025a-ff27-4f65-9202-4c6ca0f079bb admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:56.250912 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95be025a-ff27-4f65-9202-4c6ca0f079bb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:56.252532 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: WARNING py.warnings [None req-95be025a-ff27-4f65-9202-4c6ca0f079bb admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 12 17:38:56.252532 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: warnings.warn(msg) Jul 12 17:38:56.252532 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: Jul 12 17:38:56.253742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95be025a-ff27-4f65-9202-4c6ca0f079bb admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:56.363216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 776/1563] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:38:56 2022] POST /identity/v3/users => generated 290 bytes in 215 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:38:56.364522 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c288a5-a9a5-4f08-a13e-f439036de96e None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:56.364820 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c288a5-a9a5-4f08-a13e-f439036de96e None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:56.365108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c288a5-a9a5-4f08-a13e-f439036de96e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:56.385008 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-f3c288a5-a9a5-4f08-a13e-f439036de96e None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:38:56.386921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-f3c288a5-a9a5-4f08-a13e-f439036de96e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:38:56.394252 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 788/1564] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:38:56 2022] POST /identity/v3/users => generated 287 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:38:56.404604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95be025a-ff27-4f65-9202-4c6ca0f079bb admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:56.404836 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 777/1565] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:38:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:38:56.404836 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3837d45-b6d1-4cae-a7fd-3a334232077d None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:56.404836 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3837d45-b6d1-4cae-a7fd-3a334232077d None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:56.404836 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3837d45-b6d1-4cae-a7fd-3a334232077d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:56.407699 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95be025a-ff27-4f65-9202-4c6ca0f079bb admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:56.413489 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b33d03-272f-43b9-86c1-bbe9722cfd1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:56.414156 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b33d03-272f-43b9-86c1-bbe9722cfd1b admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:56.414467 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b33d03-272f-43b9-86c1-bbe9722cfd1b admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:56.414762 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b33d03-272f-43b9-86c1-bbe9722cfd1b admin admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:56.415527 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b33d03-272f-43b9-86c1-bbe9722cfd1b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:38:56.415850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b33d03-272f-43b9-86c1-bbe9722cfd1b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:56.417519 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b33d03-272f-43b9-86c1-bbe9722cfd1b admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:56.425286 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-f3837d45-b6d1-4cae-a7fd-3a334232077d None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:38:56.427349 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-f3837d45-b6d1-4cae-a7fd-3a334232077d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:38:56.440485 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 789/1566] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:38:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:38:56.440485 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5178190d-cb52-4abe-9347-560bf1565eb0 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:56.440485 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5178190d-cb52-4abe-9347-560bf1565eb0 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:56.440485 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5178190d-cb52-4abe-9347-560bf1565eb0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:56.446849 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 778/1567] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:38:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:38:56.446849 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3837d45-b6d1-4cae-a7fd-3a334232077d None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:56.460367 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3837d45-b6d1-4cae-a7fd-3a334232077d None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:56.465673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87678b7b-288d-455c-9745-8a166a4e89e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:56.466373 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87678b7b-288d-455c-9745-8a166a4e89e3 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:56.466730 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87678b7b-288d-455c-9745-8a166a4e89e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:56.466978 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87678b7b-288d-455c-9745-8a166a4e89e3 admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:56.469157 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87678b7b-288d-455c-9745-8a166a4e89e3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:56.480359 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87678b7b-288d-455c-9745-8a166a4e89e3 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:56.480644 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-5178190d-cb52-4abe-9347-560bf1565eb0 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:38:56.480644 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-5178190d-cb52-4abe-9347-560bf1565eb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:38:56.484193 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 790/1568] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:38:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:38:56.485154 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5178190d-cb52-4abe-9347-560bf1565eb0 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:56.497446 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5178190d-cb52-4abe-9347-560bf1565eb0 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:56.514490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e718202-7db1-4440-8d74-1bde8377ae92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:56.514490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e718202-7db1-4440-8d74-1bde8377ae92 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:56.514490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e718202-7db1-4440-8d74-1bde8377ae92 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:56.514490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e718202-7db1-4440-8d74-1bde8377ae92 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:56.514949 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e718202-7db1-4440-8d74-1bde8377ae92 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:56.514949 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e718202-7db1-4440-8d74-1bde8377ae92 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:56.525000 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 791/1569] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:38:56 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:38:56.526175 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e718202-7db1-4440-8d74-1bde8377ae92 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:56.540673 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e718202-7db1-4440-8d74-1bde8377ae92 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:56.545797 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936e2399-3ff8-47bf-812b-61d63141e245 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:56.546517 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936e2399-3ff8-47bf-812b-61d63141e245 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:56.546810 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936e2399-3ff8-47bf-812b-61d63141e245 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:56.547093 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936e2399-3ff8-47bf-812b-61d63141e245 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:56.550948 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-936e2399-3ff8-47bf-812b-61d63141e245 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:56.552646 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-936e2399-3ff8-47bf-812b-61d63141e245 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:56.565722 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 792/1570] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:38:56 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:38:56.566822 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936e2399-3ff8-47bf-812b-61d63141e245 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:56.582579 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936e2399-3ff8-47bf-812b-61d63141e245 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:56.588266 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14e51ec-94b4-4eec-91bc-1be6680380bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:56.589415 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14e51ec-94b4-4eec-91bc-1be6680380bf admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:56.589876 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14e51ec-94b4-4eec-91bc-1be6680380bf admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:56.590257 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14e51ec-94b4-4eec-91bc-1be6680380bf admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:56.591277 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14e51ec-94b4-4eec-91bc-1be6680380bf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:56.593225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14e51ec-94b4-4eec-91bc-1be6680380bf admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:56.598682 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 779/1571] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:38:56 2022] POST /identity/v3/users => generated 292 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:38:56.599803 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87678b7b-288d-455c-9745-8a166a4e89e3 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:56.609862 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87678b7b-288d-455c-9745-8a166a4e89e3 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:56.610322 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 793/1572] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:38:56 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:38:56.612452 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14e51ec-94b4-4eec-91bc-1be6680380bf admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:56.618107 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be00bbd5-e997-47be-a034-5f6523e07c55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:56.622208 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be00bbd5-e997-47be-a034-5f6523e07c55 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:56.622695 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be00bbd5-e997-47be-a034-5f6523e07c55 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:56.622985 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be00bbd5-e997-47be-a034-5f6523e07c55 admin admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:56.623804 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be00bbd5-e997-47be-a034-5f6523e07c55 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:38:56.624120 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be00bbd5-e997-47be-a034-5f6523e07c55 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:56.625839 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be00bbd5-e997-47be-a034-5f6523e07c55 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:56.626064 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14e51ec-94b4-4eec-91bc-1be6680380bf admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:56.632416 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1bf634-613b-4cdd-8682-3af1d1e683ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:56.633097 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1bf634-613b-4cdd-8682-3af1d1e683ee admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:56.633393 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1bf634-613b-4cdd-8682-3af1d1e683ee admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:56.633675 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1bf634-613b-4cdd-8682-3af1d1e683ee admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:56.634675 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1bf634-613b-4cdd-8682-3af1d1e683ee admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:56.636658 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1bf634-613b-4cdd-8682-3af1d1e683ee admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:56.641678 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 780/1573] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:38:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:38:56.643341 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be00bbd5-e997-47be-a034-5f6523e07c55 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:56.654771 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be00bbd5-e997-47be-a034-5f6523e07c55 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:56.660682 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450aad3b-f84a-4a21-bbcf-8f933fb61693 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:56.661517 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450aad3b-f84a-4a21-bbcf-8f933fb61693 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:56.662174 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450aad3b-f84a-4a21-bbcf-8f933fb61693 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:56.662174 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450aad3b-f84a-4a21-bbcf-8f933fb61693 admin admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:56.662885 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-450aad3b-f84a-4a21-bbcf-8f933fb61693 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:38:56.663197 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-450aad3b-f84a-4a21-bbcf-8f933fb61693 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:56.664839 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-450aad3b-f84a-4a21-bbcf-8f933fb61693 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:56.686345 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 781/1574] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:38:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:38:56.686345 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450aad3b-f84a-4a21-bbcf-8f933fb61693 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:56.691967 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450aad3b-f84a-4a21-bbcf-8f933fb61693 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:56.697864 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c894a316-b74d-47a8-8676-fcfb9a4f1ec1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:56.698724 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c894a316-b74d-47a8-8676-fcfb9a4f1ec1 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:56.699013 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c894a316-b74d-47a8-8676-fcfb9a4f1ec1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:56.699329 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c894a316-b74d-47a8-8676-fcfb9a4f1ec1 admin admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:56.700218 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c894a316-b74d-47a8-8676-fcfb9a4f1ec1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:38:56.700674 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c894a316-b74d-47a8-8676-fcfb9a4f1ec1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:56.702447 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c894a316-b74d-47a8-8676-fcfb9a4f1ec1 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:56.718563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 782/1575] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:38:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:38:56.721300 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c894a316-b74d-47a8-8676-fcfb9a4f1ec1 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:56.731375 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c894a316-b74d-47a8-8676-fcfb9a4f1ec1 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:56.738555 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95044c4-798f-4f15-804f-2f3a9233a6e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:56.740285 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95044c4-798f-4f15-804f-2f3a9233a6e6 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:56.740820 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95044c4-798f-4f15-804f-2f3a9233a6e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:56.741326 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95044c4-798f-4f15-804f-2f3a9233a6e6 admin admin] PATH_INFO: `/v3/projects/497c73f9a78743e9b24b2b6ff47d7ede/users/ce09df98c929416e8ca67b0728349bfb/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:56.770256 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e95044c4-798f-4f15-804f-2f3a9233a6e6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=497c73f9a78743e9b24b2b6ff47d7ede, user_id=ce09df98c929416e8ca67b0728349bfb, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:56.771340 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: WARNING py.warnings [None req-e95044c4-798f-4f15-804f-2f3a9233a6e6 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 12 17:38:56.771340 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: warnings.warn(msg) Jul 12 17:38:56.771340 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: Jul 12 17:38:56.773464 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e95044c4-798f-4f15-804f-2f3a9233a6e6 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:56.776772 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 794/1576] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:38:56 2022] POST /identity/v3/projects => generated 387 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:38:56.777974 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1bf634-613b-4cdd-8682-3af1d1e683ee admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:56.839390 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:38:56.842887 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1bf634-613b-4cdd-8682-3af1d1e683ee admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:56.848421 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d43e17-efec-4470-87e6-ea960fc44cd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:56.849111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d43e17-efec-4470-87e6-ea960fc44cd0 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:56.849406 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d43e17-efec-4470-87e6-ea960fc44cd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:56.849694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d43e17-efec-4470-87e6-ea960fc44cd0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:56.850467 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d43e17-efec-4470-87e6-ea960fc44cd0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:56.852115 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d43e17-efec-4470-87e6-ea960fc44cd0 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:56.860820 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 783/1577] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:38:56 2022] PUT /identity/v3/projects/497c73f9a78743e9b24b2b6ff47d7ede/users/ce09df98c929416e8ca67b0728349bfb/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:38:56.862054 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95044c4-798f-4f15-804f-2f3a9233a6e6 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:56.869236 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 795/1578] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:38:56 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:38:56.870168 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d43e17-efec-4470-87e6-ea960fc44cd0 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:56.881084 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d43e17-efec-4470-87e6-ea960fc44cd0 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:56.917796 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:38:56.943496 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95044c4-798f-4f15-804f-2f3a9233a6e6 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:56.952213 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d207ad95-4dc2-4908-bbb7-f20580fbc047 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:56.952902 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d207ad95-4dc2-4908-bbb7-f20580fbc047 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:56.953193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d207ad95-4dc2-4908-bbb7-f20580fbc047 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:56.953474 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d207ad95-4dc2-4908-bbb7-f20580fbc047 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:56.954212 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d207ad95-4dc2-4908-bbb7-f20580fbc047 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:56.955064 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1866a4d-f6a5-4b82-b7eb-99adafe0a08e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:56.956500 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d207ad95-4dc2-4908-bbb7-f20580fbc047 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:56.956678 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1866a4d-f6a5-4b82-b7eb-99adafe0a08e admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:56.957015 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1866a4d-f6a5-4b82-b7eb-99adafe0a08e admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:56.957295 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1866a4d-f6a5-4b82-b7eb-99adafe0a08e admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:56.958095 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1866a4d-f6a5-4b82-b7eb-99adafe0a08e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:56.959718 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1866a4d-f6a5-4b82-b7eb-99adafe0a08e admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:56.967268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 784/1579] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:38:56 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:38:56.967907 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d207ad95-4dc2-4908-bbb7-f20580fbc047 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:56.977100 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 796/1580] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:38:56 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:38:56.977100 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1866a4d-f6a5-4b82-b7eb-99adafe0a08e admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:56.978466 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d207ad95-4dc2-4908-bbb7-f20580fbc047 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:56.984214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9373a117-f788-4064-bc0b-13ecb615a32b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:56.984727 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1866a4d-f6a5-4b82-b7eb-99adafe0a08e admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:56.985056 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9373a117-f788-4064-bc0b-13ecb615a32b admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:56.985448 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9373a117-f788-4064-bc0b-13ecb615a32b admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:56.985835 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9373a117-f788-4064-bc0b-13ecb615a32b admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:56.986854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9373a117-f788-4064-bc0b-13ecb615a32b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:56.991840 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9373a117-f788-4064-bc0b-13ecb615a32b admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:56.991961 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f731fea3-1e00-4b4d-9cfa-7376534f9a90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:56.991961 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f731fea3-1e00-4b4d-9cfa-7376534f9a90 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:56.991961 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f731fea3-1e00-4b4d-9cfa-7376534f9a90 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:56.991961 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f731fea3-1e00-4b4d-9cfa-7376534f9a90 admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:56.992365 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f731fea3-1e00-4b4d-9cfa-7376534f9a90 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:56.993745 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f731fea3-1e00-4b4d-9cfa-7376534f9a90 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:57.088633 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 785/1581] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:38:56 2022] POST /identity/v3/projects => generated 387 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:38:57.090411 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9373a117-f788-4064-bc0b-13ecb615a32b admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:57.115168 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 797/1582] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:38:56 2022] POST /identity/v3/projects => generated 379 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:38:57.116218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f731fea3-1e00-4b4d-9cfa-7376534f9a90 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:57.177700 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9373a117-f788-4064-bc0b-13ecb615a32b admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:57.204625 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f731fea3-1e00-4b4d-9cfa-7376534f9a90 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:57.210266 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8395ca26-e345-4d25-b612-084808c3f114 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:57.210989 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8395ca26-e345-4d25-b612-084808c3f114 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:57.211279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8395ca26-e345-4d25-b612-084808c3f114 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:57.211553 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8395ca26-e345-4d25-b612-084808c3f114 admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:57.212254 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8395ca26-e345-4d25-b612-084808c3f114 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:57.214216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8395ca26-e345-4d25-b612-084808c3f114 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:57.245763 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb60dfff-4ed2-4236-bac8-fa2837acb155 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:57.245763 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb60dfff-4ed2-4236-bac8-fa2837acb155 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:57.246128 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb60dfff-4ed2-4236-bac8-fa2837acb155 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:57.246128 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb60dfff-4ed2-4236-bac8-fa2837acb155 admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:57.246921 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb60dfff-4ed2-4236-bac8-fa2837acb155 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:57.248873 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb60dfff-4ed2-4236-bac8-fa2837acb155 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:57.310185 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 798/1583] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:38:57 2022] POST /identity/v3/users => generated 294 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:38:57.310185 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745454d3-166c-4407-a2b2-6b9d8d2e4a84 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:57.310185 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745454d3-166c-4407-a2b2-6b9d8d2e4a84 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:57.310185 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745454d3-166c-4407-a2b2-6b9d8d2e4a84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:57.328744 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-745454d3-166c-4407-a2b2-6b9d8d2e4a84 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:38:57.329988 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-745454d3-166c-4407-a2b2-6b9d8d2e4a84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:38:57.334083 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 786/1584] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:38:57 2022] POST /identity/v3/projects => generated 403 bytes in 245 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:38:57.341182 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb60dfff-4ed2-4236-bac8-fa2837acb155 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:57.341337 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 799/1585] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:38:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:38:57.341337 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745454d3-166c-4407-a2b2-6b9d8d2e4a84 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:57.349903 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745454d3-166c-4407-a2b2-6b9d8d2e4a84 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:57.390304 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:38:57.403868 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:38:57.417856 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb60dfff-4ed2-4236-bac8-fa2837acb155 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:57.417856 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d233b9-42ee-4a70-8fd5-d75f769b8f78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:57.417856 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d233b9-42ee-4a70-8fd5-d75f769b8f78 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:57.418379 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d59c520-f786-446c-a452-bf361f5699af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:57.418729 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d233b9-42ee-4a70-8fd5-d75f769b8f78 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:57.418729 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d233b9-42ee-4a70-8fd5-d75f769b8f78 admin admin] PATH_INFO: `/v3/projects/33a7a8fa558b472da92a52cdd63bdbad/users/cd79a28d8f774c1fac44185df36cd24f/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:57.419015 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d59c520-f786-446c-a452-bf361f5699af admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:57.419015 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d59c520-f786-446c-a452-bf361f5699af admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:57.419015 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d59c520-f786-446c-a452-bf361f5699af admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:57.419015 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d59c520-f786-446c-a452-bf361f5699af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:57.419015 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d59c520-f786-446c-a452-bf361f5699af admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:57.430824 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 800/1586] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:38:57 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:38:57.431717 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d59c520-f786-446c-a452-bf361f5699af admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:57.438841 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d233b9-42ee-4a70-8fd5-d75f769b8f78 admin admin] RBAC: Authorizing `identity:create_grant(project_id=33a7a8fa558b472da92a52cdd63bdbad, user_id=cd79a28d8f774c1fac44185df36cd24f, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:57.443427 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d59c520-f786-446c-a452-bf361f5699af admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:57.446483 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d233b9-42ee-4a70-8fd5-d75f769b8f78 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:57.449398 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7154979d-fb58-4fa0-b703-4c5eb427e39d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:57.450668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7154979d-fb58-4fa0-b703-4c5eb427e39d admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:57.451096 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7154979d-fb58-4fa0-b703-4c5eb427e39d admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:57.451583 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7154979d-fb58-4fa0-b703-4c5eb427e39d admin admin] PATH_INFO: `/v3/projects/291ab35011594a95b495a19d62d15077/users/bb1038d2409e4f8588edee88a3f8d89d/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:57.473988 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7154979d-fb58-4fa0-b703-4c5eb427e39d admin admin] RBAC: Authorizing `identity:create_grant(project_id=291ab35011594a95b495a19d62d15077, user_id=bb1038d2409e4f8588edee88a3f8d89d, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:57.476803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7154979d-fb58-4fa0-b703-4c5eb427e39d admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:57.552231 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 787/1587] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:38:57 2022] PUT /identity/v3/projects/33a7a8fa558b472da92a52cdd63bdbad/users/cd79a28d8f774c1fac44185df36cd24f/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:38:57.552982 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d233b9-42ee-4a70-8fd5-d75f769b8f78 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:57.584236 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 801/1588] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:38:57 2022] PUT /identity/v3/projects/291ab35011594a95b495a19d62d15077/users/bb1038d2409e4f8588edee88a3f8d89d/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:38:57.593060 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7154979d-fb58-4fa0-b703-4c5eb427e39d admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:57.654972 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d233b9-42ee-4a70-8fd5-d75f769b8f78 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:57.668182 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7154979d-fb58-4fa0-b703-4c5eb427e39d admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:57.672793 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188c3c1b-12f7-4701-9ec9-7cff8c3a277a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:57.673459 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188c3c1b-12f7-4701-9ec9-7cff8c3a277a admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:57.673807 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188c3c1b-12f7-4701-9ec9-7cff8c3a277a admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:57.674741 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-188c3c1b-12f7-4701-9ec9-7cff8c3a277a admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:57.675710 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-188c3c1b-12f7-4701-9ec9-7cff8c3a277a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:57.677645 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-188c3c1b-12f7-4701-9ec9-7cff8c3a277a admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:57.730618 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e13cf9f3-2826-4ef2-8310-e771f4d061f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:57.731612 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13cf9f3-2826-4ef2-8310-e771f4d061f7 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:57.731911 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13cf9f3-2826-4ef2-8310-e771f4d061f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:57.732195 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e13cf9f3-2826-4ef2-8310-e771f4d061f7 admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:57.732917 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e13cf9f3-2826-4ef2-8310-e771f4d061f7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:57.734940 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e13cf9f3-2826-4ef2-8310-e771f4d061f7 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:57.752075 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 802/1589] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:38:57 2022] POST /identity/v3/users => generated 290 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:38:57.753284 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188c3c1b-12f7-4701-9ec9-7cff8c3a277a admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:57.765680 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-188c3c1b-12f7-4701-9ec9-7cff8c3a277a admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:57.770754 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef72523-cda4-4459-9878-065f4ab50838 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:57.771447 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef72523-cda4-4459-9878-065f4ab50838 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:57.771737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef72523-cda4-4459-9878-065f4ab50838 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:57.772019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef72523-cda4-4459-9878-065f4ab50838 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:57.772774 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef72523-cda4-4459-9878-065f4ab50838 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:57.774400 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef72523-cda4-4459-9878-065f4ab50838 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:57.786170 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 803/1590] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:38:57 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:38:57.787049 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef72523-cda4-4459-9878-065f4ab50838 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:57.797905 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef72523-cda4-4459-9878-065f4ab50838 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:57.803887 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128be50b-762c-47a7-b23f-2d0bb2e06750 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:57.804496 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128be50b-762c-47a7-b23f-2d0bb2e06750 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:57.804780 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128be50b-762c-47a7-b23f-2d0bb2e06750 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:57.805064 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128be50b-762c-47a7-b23f-2d0bb2e06750 admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:57.805776 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128be50b-762c-47a7-b23f-2d0bb2e06750 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:57.807750 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128be50b-762c-47a7-b23f-2d0bb2e06750 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:57.847943 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 788/1591] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:38:57 2022] POST /identity/v3/users => generated 294 bytes in 295 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:38:57.848793 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e13cf9f3-2826-4ef2-8310-e771f4d061f7 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:57.850559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-e13cf9f3-2826-4ef2-8310-e771f4d061f7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:38:57.871485 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 804/1592] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:38:57 2022] POST /identity/v3/users => generated 302 bytes in 85 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:38:57.872343 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128be50b-762c-47a7-b23f-2d0bb2e06750 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:57.889363 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128be50b-762c-47a7-b23f-2d0bb2e06750 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:57.895622 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5723f1a7-982f-4798-9a9e-8f9fa1dcfcd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:57.896599 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5723f1a7-982f-4798-9a9e-8f9fa1dcfcd9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:57.896920 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5723f1a7-982f-4798-9a9e-8f9fa1dcfcd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:57.900481 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5723f1a7-982f-4798-9a9e-8f9fa1dcfcd9 admin admin] PATH_INFO: `/v3/projects/84b840eaa59d4f73ac68926437f03d82/users/c8441a4a72154ee3bbb95af66abf2ffd/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:57.915235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5723f1a7-982f-4798-9a9e-8f9fa1dcfcd9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=84b840eaa59d4f73ac68926437f03d82, user_id=c8441a4a72154ee3bbb95af66abf2ffd, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:57.917980 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5723f1a7-982f-4798-9a9e-8f9fa1dcfcd9 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:57.918814 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e13cf9f3-2826-4ef2-8310-e771f4d061f7 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:57.923421 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bfd21d7-6542-4549-9d76-d77f27114605 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:57.924988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfd21d7-6542-4549-9d76-d77f27114605 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:57.925284 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfd21d7-6542-4549-9d76-d77f27114605 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:57.925568 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bfd21d7-6542-4549-9d76-d77f27114605 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:57.936065 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bfd21d7-6542-4549-9d76-d77f27114605 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:57.937538 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bfd21d7-6542-4549-9d76-d77f27114605 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:57.948699 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 789/1593] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:38:57 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:38:57.950068 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c699a93-789b-4a1c-8f6d-a2e7b80049e7 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:57.950363 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c699a93-789b-4a1c-8f6d-a2e7b80049e7 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:57.950681 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c699a93-789b-4a1c-8f6d-a2e7b80049e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:57.967021 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:38:57.970066 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-5c699a93-789b-4a1c-8f6d-a2e7b80049e7 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:38:57.972097 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-5c699a93-789b-4a1c-8f6d-a2e7b80049e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:38:57.980788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 790/1594] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:38:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:38:57.982095 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10f1c6d-8006-4662-a592-58ebb3e664f9 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:57.982395 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10f1c6d-8006-4662-a592-58ebb3e664f9 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:57.982725 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10f1c6d-8006-4662-a592-58ebb3e664f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:57.994745 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 805/1595] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:38:57 2022] PUT /identity/v3/projects/84b840eaa59d4f73ac68926437f03d82/users/c8441a4a72154ee3bbb95af66abf2ffd/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:38:57.995666 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5723f1a7-982f-4798-9a9e-8f9fa1dcfcd9 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:58.024384 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-e10f1c6d-8006-4662-a592-58ebb3e664f9 None None] MFA Rules not processed for user `bb1038d2409e4f8588edee88a3f8d89d`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:38:58.025763 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-e10f1c6d-8006-4662-a592-58ebb3e664f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:38:58.066835 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5723f1a7-982f-4798-9a9e-8f9fa1dcfcd9 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:58.072595 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f66f3ae-4061-4da9-ba8b-0f0dff918b7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:58.073722 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f66f3ae-4061-4da9-ba8b-0f0dff918b7f admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:58.073722 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f66f3ae-4061-4da9-ba8b-0f0dff918b7f admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:58.074097 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f66f3ae-4061-4da9-ba8b-0f0dff918b7f admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:58.074681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f66f3ae-4061-4da9-ba8b-0f0dff918b7f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:58.076323 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f66f3ae-4061-4da9-ba8b-0f0dff918b7f admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:58.077941 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:38:58.092174 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 806/1596] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:38:57 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:38:58.092174 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f66f3ae-4061-4da9-ba8b-0f0dff918b7f admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:58.101195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f66f3ae-4061-4da9-ba8b-0f0dff918b7f admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:58.106090 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9a015f-43a8-4d68-93d5-dab76540165c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:58.106800 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9a015f-43a8-4d68-93d5-dab76540165c admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:58.112897 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9a015f-43a8-4d68-93d5-dab76540165c admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:58.112897 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a9a015f-43a8-4d68-93d5-dab76540165c admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:58.112897 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9a015f-43a8-4d68-93d5-dab76540165c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:58.112897 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a9a015f-43a8-4d68-93d5-dab76540165c admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:58.121095 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 807/1597] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:38:58 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:38:58.121890 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9a015f-43a8-4d68-93d5-dab76540165c admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:58.127489 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 791/1598] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:38:57 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:38:58.128959 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10f1c6d-8006-4662-a592-58ebb3e664f9 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:58.155534 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a9a015f-43a8-4d68-93d5-dab76540165c admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:58.160549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10f1c6d-8006-4662-a592-58ebb3e664f9 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:58.166234 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422b19c9-e644-4c24-987c-7f7416147b4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:58.170646 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422b19c9-e644-4c24-987c-7f7416147b4e admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:58.170646 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422b19c9-e644-4c24-987c-7f7416147b4e admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:58.170646 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422b19c9-e644-4c24-987c-7f7416147b4e admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:58.170646 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-422b19c9-e644-4c24-987c-7f7416147b4e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:58.170646 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-422b19c9-e644-4c24-987c-7f7416147b4e admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:58.171140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bae1fc2-949a-4075-923e-0145f43a456e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:58.172904 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bae1fc2-949a-4075-923e-0145f43a456e admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:58.173610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bae1fc2-949a-4075-923e-0145f43a456e admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:58.174144 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bae1fc2-949a-4075-923e-0145f43a456e admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:58.175699 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bae1fc2-949a-4075-923e-0145f43a456e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:58.179401 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bae1fc2-949a-4075-923e-0145f43a456e admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:58.183637 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 808/1599] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:38:58 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:38:58.184836 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422b19c9-e644-4c24-987c-7f7416147b4e admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:58.193050 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 792/1600] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:38:58 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:38:58.200042 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85af3bb1-b2df-43ad-a28c-3ff44730a2cc None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:58.200042 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85af3bb1-b2df-43ad-a28c-3ff44730a2cc None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:58.200042 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85af3bb1-b2df-43ad-a28c-3ff44730a2cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:58.216728 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-85af3bb1-b2df-43ad-a28c-3ff44730a2cc None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:38:58.218190 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-85af3bb1-b2df-43ad-a28c-3ff44730a2cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:38:58.229287 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 793/1601] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:38:58 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:38:58.229287 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85af3bb1-b2df-43ad-a28c-3ff44730a2cc None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:58.246782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85af3bb1-b2df-43ad-a28c-3ff44730a2cc None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:58.251911 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-140fe30b-e851-415c-807c-0fddcee10d31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:58.253000 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-140fe30b-e851-415c-807c-0fddcee10d31 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:58.253000 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-140fe30b-e851-415c-807c-0fddcee10d31 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:58.253375 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-140fe30b-e851-415c-807c-0fddcee10d31 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:58.254259 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-140fe30b-e851-415c-807c-0fddcee10d31 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:58.255980 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-140fe30b-e851-415c-807c-0fddcee10d31 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:58.269435 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 794/1602] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:38:58 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:38:58.273740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-140fe30b-e851-415c-807c-0fddcee10d31 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:58.285572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422b19c9-e644-4c24-987c-7f7416147b4e admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:58.288137 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-140fe30b-e851-415c-807c-0fddcee10d31 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:58.291481 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c81c972-5206-43d0-89fd-5754e796c344 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:58.292297 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28dc429f-8784-4fe8-93da-80b9b3af3af8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:58.292840 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c81c972-5206-43d0-89fd-5754e796c344 service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:58.293245 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28dc429f-8784-4fe8-93da-80b9b3af3af8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:58.293539 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28dc429f-8784-4fe8-93da-80b9b3af3af8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:58.293751 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c81c972-5206-43d0-89fd-5754e796c344 service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:58.294208 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c81c972-5206-43d0-89fd-5754e796c344 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:58.294328 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28dc429f-8784-4fe8-93da-80b9b3af3af8 admin admin] PATH_INFO: `/v3/projects/5790727a46aa4d0cbd2790d589f7cbe4/users/7ed7774ab883440a89f061d857ad2495/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:58.306940 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c81c972-5206-43d0-89fd-5754e796c344 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:58.308752 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c81c972-5206-43d0-89fd-5754e796c344 service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:58.315251 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28dc429f-8784-4fe8-93da-80b9b3af3af8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5790727a46aa4d0cbd2790d589f7cbe4, user_id=7ed7774ab883440a89f061d857ad2495, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:58.317855 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28dc429f-8784-4fe8-93da-80b9b3af3af8 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:58.324540 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 809/1603] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:38:58 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:38:58.326244 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ae8526-0db8-45ac-be35-3c21de81e80d None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:58.326604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ae8526-0db8-45ac-be35-3c21de81e80d None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:58.326892 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ae8526-0db8-45ac-be35-3c21de81e80d None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:58.340280 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 810/1604] 10.222.0.16 () {54 vars in 961 bytes} [Tue Jul 12 17:38:58 2022] GET /identity => generated 270 bytes in 15 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:38:58.343406 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ae8526-0db8-45ac-be35-3c21de81e80d None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:58.359606 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ae8526-0db8-45ac-be35-3c21de81e80d None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:58.367965 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53585872-c9db-4fa7-8a06-ddd9218b627a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:58.368874 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53585872-c9db-4fa7-8a06-ddd9218b627a admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:58.369304 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53585872-c9db-4fa7-8a06-ddd9218b627a admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:58.369731 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53585872-c9db-4fa7-8a06-ddd9218b627a admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:58.370706 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53585872-c9db-4fa7-8a06-ddd9218b627a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:58.372669 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53585872-c9db-4fa7-8a06-ddd9218b627a admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:58.390479 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 811/1605] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:38:58 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:38:58.390479 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53585872-c9db-4fa7-8a06-ddd9218b627a admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:58.398878 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 795/1606] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:38:58 2022] PUT /identity/v3/projects/5790727a46aa4d0cbd2790d589f7cbe4/users/7ed7774ab883440a89f061d857ad2495/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:38:58.400563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28dc429f-8784-4fe8-93da-80b9b3af3af8 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:58.412848 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53585872-c9db-4fa7-8a06-ddd9218b627a admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:58.463597 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28dc429f-8784-4fe8-93da-80b9b3af3af8 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:58.467718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f286f695-4e25-4d04-b7e3-bc1e3540f813 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:58.468521 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f286f695-4e25-4d04-b7e3-bc1e3540f813 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:58.468877 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f286f695-4e25-4d04-b7e3-bc1e3540f813 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:58.469227 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f286f695-4e25-4d04-b7e3-bc1e3540f813 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:58.479416 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecaa313e-3894-4805-92f7-7f812234daaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:58.484250 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecaa313e-3894-4805-92f7-7f812234daaf admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:58.484250 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecaa313e-3894-4805-92f7-7f812234daaf admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:58.484250 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecaa313e-3894-4805-92f7-7f812234daaf admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:58.484250 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecaa313e-3894-4805-92f7-7f812234daaf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:58.484250 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecaa313e-3894-4805-92f7-7f812234daaf admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:58.497848 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 812/1607] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:38:58 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:38:58.497848 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecaa313e-3894-4805-92f7-7f812234daaf admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:58.508227 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecaa313e-3894-4805-92f7-7f812234daaf admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:58.513816 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972cabc8-8a1b-4800-aaab-a25356080da4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:58.514878 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972cabc8-8a1b-4800-aaab-a25356080da4 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:58.515132 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972cabc8-8a1b-4800-aaab-a25356080da4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:58.515420 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972cabc8-8a1b-4800-aaab-a25356080da4 admin admin] PATH_INFO: `/v3/projects/d0efdd0349e941ef913b9260e31d2d09/users/4d8217a2573048819d35ac58c5c306a2/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:58.531402 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:38:58.538255 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972cabc8-8a1b-4800-aaab-a25356080da4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d0efdd0349e941ef913b9260e31d2d09, user_id=4d8217a2573048819d35ac58c5c306a2, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:58.541013 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972cabc8-8a1b-4800-aaab-a25356080da4 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:58.547412 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f286f695-4e25-4d04-b7e3-bc1e3540f813 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:58.549184 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f286f695-4e25-4d04-b7e3-bc1e3540f813 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:58.563041 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 796/1608] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:38:58 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:38:58.565266 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe23ff73-bc20-429d-b48b-c00f8191e8da None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:58.565577 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe23ff73-bc20-429d-b48b-c00f8191e8da None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:58.565881 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe23ff73-bc20-429d-b48b-c00f8191e8da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:58.662648 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-fe23ff73-bc20-429d-b48b-c00f8191e8da None None] MFA Rules not processed for user `9012d5b88f554dc596582a7693c2a72a`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:38:58.663165 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 813/1609] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:38:58 2022] PUT /identity/v3/projects/d0efdd0349e941ef913b9260e31d2d09/users/4d8217a2573048819d35ac58c5c306a2/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:38:58.666742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972cabc8-8a1b-4800-aaab-a25356080da4 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:58.668201 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-fe23ff73-bc20-429d-b48b-c00f8191e8da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:38:58.785805 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:38:58.841505 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972cabc8-8a1b-4800-aaab-a25356080da4 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:58.859795 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a421220d-e295-4aad-90c1-66713aeab31f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:58.861874 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a421220d-e295-4aad-90c1-66713aeab31f admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:58.862199 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a421220d-e295-4aad-90c1-66713aeab31f admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:58.862731 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a421220d-e295-4aad-90c1-66713aeab31f admin admin] PATH_INFO: `/v3/projects/49227e443aa94a438bf7954aa0398cb9/users/9feeaf257bb441b7a14b57f1f493d08f/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:58.896418 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a421220d-e295-4aad-90c1-66713aeab31f admin admin] RBAC: Authorizing `identity:create_grant(project_id=49227e443aa94a438bf7954aa0398cb9, user_id=9feeaf257bb441b7a14b57f1f493d08f, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:58.899245 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 797/1610] 10.222.0.16 () {58 vars in 1058 bytes} [Tue Jul 12 17:38:58 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 335 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:38:58.899793 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a421220d-e295-4aad-90c1-66713aeab31f admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:58.905180 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171e2f04-bc28-4120-bcfc-15e13248032c None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:58.908876 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171e2f04-bc28-4120-bcfc-15e13248032c None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:58.909420 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171e2f04-bc28-4120-bcfc-15e13248032c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:58.949967 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-171e2f04-bc28-4120-bcfc-15e13248032c None None] MFA Rules not processed for user `7ed7774ab883440a89f061d857ad2495`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:38:58.951380 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-171e2f04-bc28-4120-bcfc-15e13248032c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:38:59.029023 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 814/1611] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:38:58 2022] PUT /identity/v3/projects/49227e443aa94a438bf7954aa0398cb9/users/9feeaf257bb441b7a14b57f1f493d08f/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 365 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:38:59.030568 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a421220d-e295-4aad-90c1-66713aeab31f admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:59.059119 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 798/1612] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:38:58 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:38:59.063408 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdea270f-912e-4319-ad9c-d50d1cadb18f None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:59.063714 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdea270f-912e-4319-ad9c-d50d1cadb18f None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:59.063996 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdea270f-912e-4319-ad9c-d50d1cadb18f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:59.109065 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-cdea270f-912e-4319-ad9c-d50d1cadb18f None None] MFA Rules not processed for user `4d8217a2573048819d35ac58c5c306a2`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:38:59.111845 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-cdea270f-912e-4319-ad9c-d50d1cadb18f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:38:59.118566 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a421220d-e295-4aad-90c1-66713aeab31f admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:59.124458 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775a97df-6349-45f1-8480-2d8040a923f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:59.125365 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775a97df-6349-45f1-8480-2d8040a923f6 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:59.125653 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775a97df-6349-45f1-8480-2d8040a923f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:59.125938 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775a97df-6349-45f1-8480-2d8040a923f6 admin admin] PATH_INFO: `/v3/projects/0fa254f4eb59487a91bd5807aca4028a/users/18a8a578f3554b3dbf53a50db6c94faf/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:59.159974 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-775a97df-6349-45f1-8480-2d8040a923f6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0fa254f4eb59487a91bd5807aca4028a, user_id=18a8a578f3554b3dbf53a50db6c94faf, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:59.169909 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-775a97df-6349-45f1-8480-2d8040a923f6 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:59.198604 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:38:59.231061 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 799/1613] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:38:59 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:38:59.231815 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdea270f-912e-4319-ad9c-d50d1cadb18f None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:59.293923 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 815/1614] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:38:59 2022] PUT /identity/v3/projects/0fa254f4eb59487a91bd5807aca4028a/users/18a8a578f3554b3dbf53a50db6c94faf/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 264 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:38:59.296814 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6ea8fe-9136-4ce6-b059-2132983c3a51 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:59.297673 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6ea8fe-9136-4ce6-b059-2132983c3a51 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:59.298212 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6ea8fe-9136-4ce6-b059-2132983c3a51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:59.326545 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-9d6ea8fe-9136-4ce6-b059-2132983c3a51 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:38:59.328480 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-9d6ea8fe-9136-4ce6-b059-2132983c3a51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:38:59.362655 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdea270f-912e-4319-ad9c-d50d1cadb18f None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:59.447003 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 816/1615] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:38:59 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:38:59.448324 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6ea8fe-9136-4ce6-b059-2132983c3a51 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:59.452815 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d99b9ee-9217-4108-8980-af996cac101a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:59.453557 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d99b9ee-9217-4108-8980-af996cac101a service nova] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:59.453983 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d99b9ee-9217-4108-8980-af996cac101a service nova] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:59.454410 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d99b9ee-9217-4108-8980-af996cac101a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:59.463127 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6ea8fe-9136-4ce6-b059-2132983c3a51 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:59.473863 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8a8b99e-cd33-4e25-b975-5de793a08b21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:59.474666 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a8b99e-cd33-4e25-b975-5de793a08b21 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:59.474963 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a8b99e-cd33-4e25-b975-5de793a08b21 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:59.475251 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a8b99e-cd33-4e25-b975-5de793a08b21 admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:59.476084 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a8b99e-cd33-4e25-b975-5de793a08b21 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:59.478189 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8a8b99e-cd33-4e25-b975-5de793a08b21 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:59.550133 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d99b9ee-9217-4108-8980-af996cac101a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:59.554407 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d99b9ee-9217-4108-8980-af996cac101a service nova] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:59.576841 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 800/1616] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:38:59 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:38:59.578865 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d99b9ee-9217-4108-8980-af996cac101a service nova] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:59.596587 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d99b9ee-9217-4108-8980-af996cac101a service nova] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:59.601973 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742f6d19-1176-4c8c-a550-3fa1d9f4cb68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:59.602719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742f6d19-1176-4c8c-a550-3fa1d9f4cb68 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:59.603058 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742f6d19-1176-4c8c-a550-3fa1d9f4cb68 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:59.603380 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-742f6d19-1176-4c8c-a550-3fa1d9f4cb68 admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:59.604118 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742f6d19-1176-4c8c-a550-3fa1d9f4cb68 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:59.606266 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-742f6d19-1176-4c8c-a550-3fa1d9f4cb68 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:59.609553 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 817/1617] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:38:59 2022] POST /identity/v3/projects => generated 383 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:38:59.611379 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7262e3e9-ad25-4390-a41d-c16b3ac66352 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:59.612447 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7262e3e9-ad25-4390-a41d-c16b3ac66352 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:59.612750 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7262e3e9-ad25-4390-a41d-c16b3ac66352 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:59.645051 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-7262e3e9-ad25-4390-a41d-c16b3ac66352 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:38:59.646422 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-7262e3e9-ad25-4390-a41d-c16b3ac66352 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:38:59.666829 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:38:59.735197 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:38:59.750038 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 801/1618] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:38:59 2022] POST /identity/v3/projects => generated 377 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:38:59.754629 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742f6d19-1176-4c8c-a550-3fa1d9f4cb68 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:59.755915 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 818/1619] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:38:59 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:38:59.756607 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7262e3e9-ad25-4390-a41d-c16b3ac66352 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:59.785746 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7262e3e9-ad25-4390-a41d-c16b3ac66352 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:59.883646 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-742f6d19-1176-4c8c-a550-3fa1d9f4cb68 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:59.885904 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d8b0e4-fb82-4e13-8c54-4f5cabb7b338 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:59.886678 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d8b0e4-fb82-4e13-8c54-4f5cabb7b338 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:59.886969 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d8b0e4-fb82-4e13-8c54-4f5cabb7b338 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:59.887261 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d8b0e4-fb82-4e13-8c54-4f5cabb7b338 admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:59.887956 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21d8b0e4-fb82-4e13-8c54-4f5cabb7b338 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:59.890161 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21d8b0e4-fb82-4e13-8c54-4f5cabb7b338 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:59.891633 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00876d2-e25f-4e1e-8c71-46a13c993539 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:59.892903 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00876d2-e25f-4e1e-8c71-46a13c993539 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:59.894643 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00876d2-e25f-4e1e-8c71-46a13c993539 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:59.895018 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00876d2-e25f-4e1e-8c71-46a13c993539 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:59.908020 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f00876d2-e25f-4e1e-8c71-46a13c993539 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:59.912165 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f00876d2-e25f-4e1e-8c71-46a13c993539 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:38:59.926584 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 802/1620] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:38:59 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 172 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:38:59.928804 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00876d2-e25f-4e1e-8c71-46a13c993539 service neutron] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:38:59.941782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00876d2-e25f-4e1e-8c71-46a13c993539 service neutron] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:38:59.963684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e40b6a3-8334-4e56-9779-75758c60e461 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:38:59.963684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e40b6a3-8334-4e56-9779-75758c60e461 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:38:59.963684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e40b6a3-8334-4e56-9779-75758c60e461 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:38:59.963684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e40b6a3-8334-4e56-9779-75758c60e461 admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:38:59.964063 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e40b6a3-8334-4e56-9779-75758c60e461 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:38:59.965263 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e40b6a3-8334-4e56-9779-75758c60e461 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:00.027698 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 819/1621] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:38:59 2022] POST /identity/v3/users => generated 298 bytes in 271 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:00.028535 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d8b0e4-fb82-4e13-8c54-4f5cabb7b338 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:00.040162 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d8b0e4-fb82-4e13-8c54-4f5cabb7b338 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:00.045046 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e49998-f015-4605-83fe-416bb437fa6a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:00.045709 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e49998-f015-4605-83fe-416bb437fa6a service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:00.046051 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e49998-f015-4605-83fe-416bb437fa6a service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:00.046333 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e49998-f015-4605-83fe-416bb437fa6a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:00.058070 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93e49998-f015-4605-83fe-416bb437fa6a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:00.059643 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93e49998-f015-4605-83fe-416bb437fa6a service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:00.067450 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 803/1622] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:38:59 2022] POST /identity/v3/users => generated 289 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:00.068288 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055c246e-96f3-4549-a177-805ed421a600 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:00.068591 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055c246e-96f3-4549-a177-805ed421a600 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:00.068880 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055c246e-96f3-4549-a177-805ed421a600 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:00.072611 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 820/1623] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:00 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:39:00.073550 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e49998-f015-4605-83fe-416bb437fa6a service neutron] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:00.076393 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 804/1624] 10.222.0.16 () {54 vars in 912 bytes} [Tue Jul 12 17:39:00 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:39:00.083763 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055c246e-96f3-4549-a177-805ed421a600 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:00.085606 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e49998-f015-4605-83fe-416bb437fa6a service neutron] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:00.091184 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7e4156-2eac-49a3-9171-2a5fbd803783 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:00.091920 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7e4156-2eac-49a3-9171-2a5fbd803783 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:00.092244 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7e4156-2eac-49a3-9171-2a5fbd803783 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:00.092576 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7e4156-2eac-49a3-9171-2a5fbd803783 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:00.093429 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7e4156-2eac-49a3-9171-2a5fbd803783 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:00.095252 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7e4156-2eac-49a3-9171-2a5fbd803783 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:00.108695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 821/1625] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:00 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:00.110336 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b3154d-4261-459c-9fdd-8e88e0b5b44e None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:00.110593 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b3154d-4261-459c-9fdd-8e88e0b5b44e None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:00.110875 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b3154d-4261-459c-9fdd-8e88e0b5b44e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:00.120172 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055c246e-96f3-4549-a177-805ed421a600 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:00.125639 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96c9dbe-bca1-4960-8c25-9b9b1a8a6108 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:00.126329 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96c9dbe-bca1-4960-8c25-9b9b1a8a6108 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:00.127271 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96c9dbe-bca1-4960-8c25-9b9b1a8a6108 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:00.127749 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96c9dbe-bca1-4960-8c25-9b9b1a8a6108 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:00.128497 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96c9dbe-bca1-4960-8c25-9b9b1a8a6108 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:00.130136 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c96c9dbe-bca1-4960-8c25-9b9b1a8a6108 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:00.134311 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-71b3154d-4261-459c-9fdd-8e88e0b5b44e None None] MFA Rules not processed for user `9012d5b88f554dc596582a7693c2a72a`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:00.135814 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-71b3154d-4261-459c-9fdd-8e88e0b5b44e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:00.156497 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 805/1626] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:00 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:00.156905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96c9dbe-bca1-4960-8c25-9b9b1a8a6108 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:00.176567 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96c9dbe-bca1-4960-8c25-9b9b1a8a6108 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:00.192673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105c5e76-9fd8-44fb-8fd8-55097a68faf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:00.194037 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105c5e76-9fd8-44fb-8fd8-55097a68faf8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:00.194598 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105c5e76-9fd8-44fb-8fd8-55097a68faf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:00.195347 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105c5e76-9fd8-44fb-8fd8-55097a68faf8 admin admin] PATH_INFO: `/v3/projects/e9d33678c4554583bfa489acbfeebc29/users/99be694ce83647f9aaf519673d73dde4/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:00.223646 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 822/1627] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Jul 12 17:39:00 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:00.224658 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b3154d-4261-459c-9fdd-8e88e0b5b44e None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:00.224793 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-105c5e76-9fd8-44fb-8fd8-55097a68faf8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e9d33678c4554583bfa489acbfeebc29, user_id=99be694ce83647f9aaf519673d73dde4, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:00.228123 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-105c5e76-9fd8-44fb-8fd8-55097a68faf8 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:00.239272 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b3154d-4261-459c-9fdd-8e88e0b5b44e None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:00.246393 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc5cb58-d600-48d4-b36f-eac002567432 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:00.249229 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc5cb58-d600-48d4-b36f-eac002567432 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:00.249229 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc5cb58-d600-48d4-b36f-eac002567432 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:00.249229 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc5cb58-d600-48d4-b36f-eac002567432 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:00.249229 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc5cb58-d600-48d4-b36f-eac002567432 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:00.250476 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc5cb58-d600-48d4-b36f-eac002567432 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:00.267574 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 823/1628] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:00 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:00.267574 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc5cb58-d600-48d4-b36f-eac002567432 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:00.280966 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:00.326249 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 806/1629] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:39:00 2022] PUT /identity/v3/projects/e9d33678c4554583bfa489acbfeebc29/users/99be694ce83647f9aaf519673d73dde4/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:00.327418 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105c5e76-9fd8-44fb-8fd8-55097a68faf8 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:00.338324 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc5cb58-d600-48d4-b36f-eac002567432 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:00.418954 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802d4a34-fffc-4044-ba1a-c45a33be2a08 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:00.418954 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802d4a34-fffc-4044-ba1a-c45a33be2a08 service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:00.418954 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802d4a34-fffc-4044-ba1a-c45a33be2a08 service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:00.418954 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-802d4a34-fffc-4044-ba1a-c45a33be2a08 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:00.457582 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105c5e76-9fd8-44fb-8fd8-55097a68faf8 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:00.464681 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519f824c-7e14-447b-a4df-0433ede0a40b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:00.465725 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519f824c-7e14-447b-a4df-0433ede0a40b admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:00.466211 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519f824c-7e14-447b-a4df-0433ede0a40b admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:00.466757 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519f824c-7e14-447b-a4df-0433ede0a40b admin admin] PATH_INFO: `/v3/projects/95e70964673a48d092a7568dc6927b89/users/34de8ce915784775a280db6df99e1965/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:00.491645 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519f824c-7e14-447b-a4df-0433ede0a40b admin admin] RBAC: Authorizing `identity:create_grant(project_id=95e70964673a48d092a7568dc6927b89, user_id=34de8ce915784775a280db6df99e1965, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:00.495782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519f824c-7e14-447b-a4df-0433ede0a40b admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:00.542410 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-802d4a34-fffc-4044-ba1a-c45a33be2a08 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:00.543996 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-802d4a34-fffc-4044-ba1a-c45a33be2a08 service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:00.567139 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 824/1630] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:00 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:39:00.567139 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802d4a34-fffc-4044-ba1a-c45a33be2a08 service cinder] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:00.582798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-802d4a34-fffc-4044-ba1a-c45a33be2a08 service cinder] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:00.587266 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 807/1631] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:39:00 2022] PUT /identity/v3/projects/95e70964673a48d092a7568dc6927b89/users/34de8ce915784775a280db6df99e1965/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:00.597030 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba71f6c6-8e69-416e-b24c-32d00db72ccb None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:00.597402 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba71f6c6-8e69-416e-b24c-32d00db72ccb None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:00.597693 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba71f6c6-8e69-416e-b24c-32d00db72ccb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:00.647393 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-ba71f6c6-8e69-416e-b24c-32d00db72ccb None None] MFA Rules not processed for user `34de8ce915784775a280db6df99e1965`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:00.648572 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-ba71f6c6-8e69-416e-b24c-32d00db72ccb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:00.693346 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1956f977-8c9e-4391-b3ef-3e1783d70dbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:00.694220 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1956f977-8c9e-4391-b3ef-3e1783d70dbc admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:00.694883 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1956f977-8c9e-4391-b3ef-3e1783d70dbc admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:00.695709 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1956f977-8c9e-4391-b3ef-3e1783d70dbc admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:00.696705 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1956f977-8c9e-4391-b3ef-3e1783d70dbc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:00.699591 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1956f977-8c9e-4391-b3ef-3e1783d70dbc admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:00.709114 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:00.717064 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 825/1632] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:00 2022] GET /identity/v3/roles => generated 1420 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:00.721489 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1956f977-8c9e-4391-b3ef-3e1783d70dbc admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:00.739785 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1956f977-8c9e-4391-b3ef-3e1783d70dbc admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:00.753727 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23997895-9e65-422f-92dc-cc6b87608bf6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:00.756653 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23997895-9e65-422f-92dc-cc6b87608bf6 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:00.758319 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23997895-9e65-422f-92dc-cc6b87608bf6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:00.758689 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23997895-9e65-422f-92dc-cc6b87608bf6 admin admin] PATH_INFO: `/v3/projects/e9d33678c4554583bfa489acbfeebc29/users/99be694ce83647f9aaf519673d73dde4/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:00.763144 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23997895-9e65-422f-92dc-cc6b87608bf6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e9d33678c4554583bfa489acbfeebc29, user_id=99be694ce83647f9aaf519673d73dde4, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:00.773168 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23997895-9e65-422f-92dc-cc6b87608bf6 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:00.801032 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 826/1633] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:39:00 2022] PUT /identity/v3/projects/e9d33678c4554583bfa489acbfeebc29/users/99be694ce83647f9aaf519673d73dde4/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:00.811535 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad186154-a0d2-4b07-a8df-b6cb288fa9c3 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:00.811925 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad186154-a0d2-4b07-a8df-b6cb288fa9c3 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:00.812103 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad186154-a0d2-4b07-a8df-b6cb288fa9c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:00.817032 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 808/1634] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:00 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:00.828473 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba71f6c6-8e69-416e-b24c-32d00db72ccb None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:00.846962 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:00.863548 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba71f6c6-8e69-416e-b24c-32d00db72ccb None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:00.867159 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-ad186154-a0d2-4b07-a8df-b6cb288fa9c3 None None] MFA Rules not processed for user `99be694ce83647f9aaf519673d73dde4`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:00.869016 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-ad186154-a0d2-4b07-a8df-b6cb288fa9c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:01.014739 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4183c677-8553-4988-881b-fab4b2431926 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:01.015425 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4183c677-8553-4988-881b-fab4b2431926 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:01.020242 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4183c677-8553-4988-881b-fab4b2431926 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:01.020543 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4183c677-8553-4988-881b-fab4b2431926 admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:01.021245 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4183c677-8553-4988-881b-fab4b2431926 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:01.023232 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4183c677-8553-4988-881b-fab4b2431926 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:01.037877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 827/1635] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:00 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:39:01.052535 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad186154-a0d2-4b07-a8df-b6cb288fa9c3 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:01.124368 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad186154-a0d2-4b07-a8df-b6cb288fa9c3 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:01.137006 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60eedb51-9d68-458e-b977-8819a83b287a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:01.137781 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60eedb51-9d68-458e-b977-8819a83b287a service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:01.138217 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60eedb51-9d68-458e-b977-8819a83b287a service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:01.138540 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60eedb51-9d68-458e-b977-8819a83b287a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:01.151943 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60eedb51-9d68-458e-b977-8819a83b287a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:01.155084 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60eedb51-9d68-458e-b977-8819a83b287a service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:01.171558 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 828/1636] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:01 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:39:01.217405 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 809/1637] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:39:00 2022] POST /identity/v3/projects => generated 377 bytes in 389 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:01.218388 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60eedb51-9d68-458e-b977-8819a83b287a service cinder] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:01.233419 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60eedb51-9d68-458e-b977-8819a83b287a service cinder] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:01.261731 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4183c677-8553-4988-881b-fab4b2431926 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:01.385041 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b513faba-3491-495b-b0f2-e2892d073fc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:01.386055 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b513faba-3491-495b-b0f2-e2892d073fc4 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:01.386605 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b513faba-3491-495b-b0f2-e2892d073fc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:01.387075 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b513faba-3491-495b-b0f2-e2892d073fc4 admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:01.389025 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b513faba-3491-495b-b0f2-e2892d073fc4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:01.391412 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b513faba-3491-495b-b0f2-e2892d073fc4 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:01.405786 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4183c677-8553-4988-881b-fab4b2431926 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:01.415012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af510a8-aa56-46ac-b58d-20b4a3664941 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:01.415841 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af510a8-aa56-46ac-b58d-20b4a3664941 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:01.416129 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af510a8-aa56-46ac-b58d-20b4a3664941 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:01.416426 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af510a8-aa56-46ac-b58d-20b4a3664941 admin admin] PATH_INFO: `/v3/users/7ed7774ab883440a89f061d857ad2495` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:01.418315 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af510a8-aa56-46ac-b58d-20b4a3664941 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7ed7774ab883440a89f061d857ad2495)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:01.419434 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: WARNING py.warnings [None req-9af510a8-aa56-46ac-b58d-20b4a3664941 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 12 17:39:01.419434 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: warnings.warn(msg) Jul 12 17:39:01.419434 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: Jul 12 17:39:01.422863 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af510a8-aa56-46ac-b58d-20b4a3664941 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:01.611975 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 829/1638] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:39:01 2022] POST /identity/v3/users => generated 295 bytes in 304 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:01.611975 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b513faba-3491-495b-b0f2-e2892d073fc4 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:01.611975 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b513faba-3491-495b-b0f2-e2892d073fc4 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:01.611975 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd63a689-ffb9-4837-af49-efc5c72c3b2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:01.615753 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd63a689-ffb9-4837-af49-efc5c72c3b2d admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:01.615753 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd63a689-ffb9-4837-af49-efc5c72c3b2d admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:01.615753 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd63a689-ffb9-4837-af49-efc5c72c3b2d admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:01.615753 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd63a689-ffb9-4837-af49-efc5c72c3b2d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:01.615753 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd63a689-ffb9-4837-af49-efc5c72c3b2d admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:01.639849 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 830/1639] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:01 2022] GET /identity/v3/roles => generated 1420 bytes in 114 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:01.641480 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd63a689-ffb9-4837-af49-efc5c72c3b2d admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:01.661768 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:01.669350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd63a689-ffb9-4837-af49-efc5c72c3b2d admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:01.673130 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67153c9-62c6-4168-91da-a835716dbef9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:01.674440 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67153c9-62c6-4168-91da-a835716dbef9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:01.676819 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67153c9-62c6-4168-91da-a835716dbef9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:01.677191 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67153c9-62c6-4168-91da-a835716dbef9 admin admin] PATH_INFO: `/v3/projects/78a7fac0a1ad4f098766b812b54647a9/users/ceac25e8a7c849c7a578c3ac762fe824/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:01.705753 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67153c9-62c6-4168-91da-a835716dbef9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=78a7fac0a1ad4f098766b812b54647a9, user_id=ceac25e8a7c849c7a578c3ac762fe824, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:01.710243 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67153c9-62c6-4168-91da-a835716dbef9 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:01.749841 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-9af510a8-aa56-46ac-b58d-20b4a3664941 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7ed7774ab883440a89f061d857ad2495'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:01.833674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 831/1640] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:39:01 2022] PUT /identity/v3/projects/78a7fac0a1ad4f098766b812b54647a9/users/ceac25e8a7c849c7a578c3ac762fe824/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:01.840212 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-9af510a8-aa56-46ac-b58d-20b4a3664941 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7ed7774ab883440a89f061d857ad2495'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:01.841693 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-9af510a8-aa56-46ac-b58d-20b4a3664941 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7ed7774ab883440a89f061d857ad2495'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:01.848990 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67153c9-62c6-4168-91da-a835716dbef9 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:01.853409 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-c67153c9-62c6-4168-91da-a835716dbef9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:01.860494 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-9af510a8-aa56-46ac-b58d-20b4a3664941 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7ed7774ab883440a89f061d857ad2495'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:01.891640 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-9af510a8-aa56-46ac-b58d-20b4a3664941 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7ed7774ab883440a89f061d857ad2495'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:01.924803 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 810/1641] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:39:01 2022] DELETE /identity/v3/users/7ed7774ab883440a89f061d857ad2495 => generated 0 bytes in 664 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:39:01.932663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec758e6b-9c78-4905-9468-6ead6b4da358 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:01.932663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec758e6b-9c78-4905-9468-6ead6b4da358 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:01.933008 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec758e6b-9c78-4905-9468-6ead6b4da358 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:01.969939 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-ec758e6b-9c78-4905-9468-6ead6b4da358 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:01.974087 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-ec758e6b-9c78-4905-9468-6ead6b4da358 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:02.006885 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67153c9-62c6-4168-91da-a835716dbef9 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:02.410036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=92,keystone:INSERT=1,keystone:DELETE=7 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:02.410036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 811/1642] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 231 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:02.410036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec758e6b-9c78-4905-9468-6ead6b4da358 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:02.410036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-ec758e6b-9c78-4905-9468-6ead6b4da358 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:02.416184 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c8c380-d2f3-435f-888f-a2ffa8271ddf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:02.416184 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c8c380-d2f3-435f-888f-a2ffa8271ddf admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:02.416184 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c8c380-d2f3-435f-888f-a2ffa8271ddf admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:02.416184 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c8c380-d2f3-435f-888f-a2ffa8271ddf admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:02.416947 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c8c380-d2f3-435f-888f-a2ffa8271ddf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:02.416947 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7c8c380-d2f3-435f-888f-a2ffa8271ddf admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:02.416947 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 832/1643] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:01 2022] GET /identity/v3/roles => generated 1420 bytes in 419 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:02.416947 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c8c380-d2f3-435f-888f-a2ffa8271ddf admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:02.416947 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c8c380-d2f3-435f-888f-a2ffa8271ddf admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:02.417343 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adacdb2d-4897-43a3-8f90-0b74eacf6484 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:02.417343 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adacdb2d-4897-43a3-8f90-0b74eacf6484 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:02.417343 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adacdb2d-4897-43a3-8f90-0b74eacf6484 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:02.417343 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adacdb2d-4897-43a3-8f90-0b74eacf6484 admin admin] PATH_INFO: `/v3/projects/78a7fac0a1ad4f098766b812b54647a9/users/ceac25e8a7c849c7a578c3ac762fe824/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:02.417607 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adacdb2d-4897-43a3-8f90-0b74eacf6484 admin admin] RBAC: Authorizing `identity:create_grant(project_id=78a7fac0a1ad4f098766b812b54647a9, user_id=ceac25e8a7c849c7a578c3ac762fe824, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:02.417607 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adacdb2d-4897-43a3-8f90-0b74eacf6484 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:02.417730 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec758e6b-9c78-4905-9468-6ead6b4da358 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:02.417811 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 833/1644] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:39:02 2022] PUT /identity/v3/projects/78a7fac0a1ad4f098766b812b54647a9/users/ceac25e8a7c849c7a578c3ac762fe824/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:02.417811 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165dd173-e158-4926-b64b-786726c5ed12 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:02.417811 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165dd173-e158-4926-b64b-786726c5ed12 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:02.417811 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165dd173-e158-4926-b64b-786726c5ed12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:02.466790 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb8757e2-879c-463a-8183-c9c0d64f05dd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:02.467891 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8757e2-879c-463a-8183-c9c0d64f05dd service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:02.468180 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8757e2-879c-463a-8183-c9c0d64f05dd service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:02.468606 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8757e2-879c-463a-8183-c9c0d64f05dd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:02.484528 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-165dd173-e158-4926-b64b-786726c5ed12 None None] MFA Rules not processed for user `ceac25e8a7c849c7a578c3ac762fe824`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:02.486511 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-165dd173-e158-4926-b64b-786726c5ed12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:02.692988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb8757e2-879c-463a-8183-c9c0d64f05dd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:02.700302 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb8757e2-879c-463a-8183-c9c0d64f05dd service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:02.742905 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 834/1645] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:02 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 351 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:02.754686 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165dd173-e158-4926-b64b-786726c5ed12 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:02.756235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-165dd173-e158-4926-b64b-786726c5ed12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:02.774954 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 812/1646] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:02 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 577 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:39:02.942406 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165dd173-e158-4926-b64b-786726c5ed12 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:02.953780 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e80930-e21d-4e48-80a0-40d569ef4c96 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:02.955128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e80930-e21d-4e48-80a0-40d569ef4c96 service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:02.955557 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e80930-e21d-4e48-80a0-40d569ef4c96 service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:02.955801 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e80930-e21d-4e48-80a0-40d569ef4c96 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:02.979928 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e80930-e21d-4e48-80a0-40d569ef4c96 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:02.982772 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e80930-e21d-4e48-80a0-40d569ef4c96 service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:03.019678 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 835/1647] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:02 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 265 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:39:03.135058 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ed5917-fbd5-443c-ade0-5471805fd543 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:03.136134 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ed5917-fbd5-443c-ade0-5471805fd543 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:03.144864 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ed5917-fbd5-443c-ade0-5471805fd543 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:03.144864 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 813/1648] 10.222.0.16 () {54 vars in 961 bytes} [Tue Jul 12 17:39:03 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:39:03.156476 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed5e445-fbe4-44ff-a51a-b0353bbaab43 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:03.156710 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed5e445-fbe4-44ff-a51a-b0353bbaab43 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:03.156831 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed5e445-fbe4-44ff-a51a-b0353bbaab43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:03.192959 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-3ed5e445-fbe4-44ff-a51a-b0353bbaab43 None None] MFA Rules not processed for user `9012d5b88f554dc596582a7693c2a72a`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:03.194210 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-3ed5e445-fbe4-44ff-a51a-b0353bbaab43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:03.268704 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:03.290757 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ed5917-fbd5-443c-ade0-5471805fd543 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:03.294052 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-c8ed5917-fbd5-443c-ade0-5471805fd543 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:03.324982 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8ed5917-fbd5-443c-ade0-5471805fd543 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:03.332053 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6e9d79-ff58-43c5-b839-77c4b07c4b14 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:03.333033 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6e9d79-ff58-43c5-b839-77c4b07c4b14 service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:03.333469 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6e9d79-ff58-43c5-b839-77c4b07c4b14 service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:03.333957 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6e9d79-ff58-43c5-b839-77c4b07c4b14 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:03.336516 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-3f6e9d79-ff58-43c5-b839-77c4b07c4b14 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:03.482355 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 836/1649] 10.222.0.16 () {58 vars in 1058 bytes} [Tue Jul 12 17:39:03 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 327 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:03.521492 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f6e9d79-ff58-43c5-b839-77c4b07c4b14 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:03.523186 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f6e9d79-ff58-43c5-b839-77c4b07c4b14 service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:03.526885 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0ae189-0c6c-4f3d-9c7b-aea5fbab5f53 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:03.527691 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0ae189-0c6c-4f3d-9c7b-aea5fbab5f53 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:03.528492 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0ae189-0c6c-4f3d-9c7b-aea5fbab5f53 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:03.533515 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 837/1650] 10.222.0.16 () {54 vars in 912 bytes} [Tue Jul 12 17:39:03 2022] GET /identity => generated 270 bytes in 14 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:39:03.543804 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d625101-5051-4ef4-9c02-71ae32742806 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:03.548033 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d625101-5051-4ef4-9c02-71ae32742806 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:03.549717 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 814/1651] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:03 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 260 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:39:03.550061 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d625101-5051-4ef4-9c02-71ae32742806 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:03.624822 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-9d625101-5051-4ef4-9c02-71ae32742806 None None] MFA Rules not processed for user `9012d5b88f554dc596582a7693c2a72a`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:03.626023 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-9d625101-5051-4ef4-9c02-71ae32742806 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:03.666767 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 838/1652] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Jul 12 17:39:03 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:03.683298 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6e9d79-ff58-43c5-b839-77c4b07c4b14 service cinder] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:03.693271 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6e9d79-ff58-43c5-b839-77c4b07c4b14 service cinder] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:03.708647 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759a4f56-87c8-4e2a-b2f4-a771f5d9eb42 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:03.713369 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759a4f56-87c8-4e2a-b2f4-a771f5d9eb42 service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:03.714158 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759a4f56-87c8-4e2a-b2f4-a771f5d9eb42 service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:03.716106 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759a4f56-87c8-4e2a-b2f4-a771f5d9eb42 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:03.748556 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759a4f56-87c8-4e2a-b2f4-a771f5d9eb42 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:03.750357 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759a4f56-87c8-4e2a-b2f4-a771f5d9eb42 service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:03.772028 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 815/1653] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:03 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:39:04.515352 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:INSERT=1,keystone:DELETE=3,keystone:SELECT=96 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:04.529156 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d625101-5051-4ef4-9c02-71ae32742806 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:04.530782 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-9d625101-5051-4ef4-9c02-71ae32742806 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:04.558727 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d625101-5051-4ef4-9c02-71ae32742806 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:04.583635 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c10f4bb-cbc8-43fb-aa3e-8404ff57ab96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:04.586472 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c10f4bb-cbc8-43fb-aa3e-8404ff57ab96 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:04.587744 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c10f4bb-cbc8-43fb-aa3e-8404ff57ab96 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:04.588811 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c10f4bb-cbc8-43fb-aa3e-8404ff57ab96 admin admin] PATH_INFO: `/v3/projects/5790727a46aa4d0cbd2790d589f7cbe4` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:04.593370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c10f4bb-cbc8-43fb-aa3e-8404ff57ab96 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5790727a46aa4d0cbd2790d589f7cbe4)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:04.594478 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: WARNING py.warnings [None req-1c10f4bb-cbc8-43fb-aa3e-8404ff57ab96 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 12 17:39:04.594478 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: warnings.warn(msg) Jul 12 17:39:04.594478 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: Jul 12 17:39:04.596865 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c10f4bb-cbc8-43fb-aa3e-8404ff57ab96 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:04.865547 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-1c10f4bb-cbc8-43fb-aa3e-8404ff57ab96 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5790727a46aa4d0cbd2790d589f7cbe4', 'request_id': 'req-1c10f4bb-cbc8-43fb-aa3e-8404ff57ab96'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:04.877339 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-1c10f4bb-cbc8-43fb-aa3e-8404ff57ab96 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5790727a46aa4d0cbd2790d589f7cbe4', 'request_id': 'req-1c10f4bb-cbc8-43fb-aa3e-8404ff57ab96'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:04.968203 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-1c10f4bb-cbc8-43fb-aa3e-8404ff57ab96 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5790727a46aa4d0cbd2790d589f7cbe4', 'request_id': 'req-1c10f4bb-cbc8-43fb-aa3e-8404ff57ab96'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:04.993627 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:04.999179 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-1c10f4bb-cbc8-43fb-aa3e-8404ff57ab96 admin admin] The token cache is being invalidate because project 5790727a46aa4d0cbd2790d589f7cbe4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:39:04.999482 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-1c10f4bb-cbc8-43fb-aa3e-8404ff57ab96 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:05.000408 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-1c10f4bb-cbc8-43fb-aa3e-8404ff57ab96 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:05.006648 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 839/1654] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:39:04 2022] DELETE /identity/v3/projects/5790727a46aa4d0cbd2790d589f7cbe4 => generated 0 bytes in 477 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:05.018811 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759a4f56-87c8-4e2a-b2f4-a771f5d9eb42 service cinder] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:05.034869 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759a4f56-87c8-4e2a-b2f4-a771f5d9eb42 service cinder] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:05.184380 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5825f4da-b55a-4760-9e6e-fc99d34bc5fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:05.185967 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5825f4da-b55a-4760-9e6e-fc99d34bc5fa admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:05.186791 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5825f4da-b55a-4760-9e6e-fc99d34bc5fa admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:05.190791 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5825f4da-b55a-4760-9e6e-fc99d34bc5fa admin admin] PATH_INFO: `/v3/users/99be694ce83647f9aaf519673d73dde4` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:05.195171 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5825f4da-b55a-4760-9e6e-fc99d34bc5fa admin admin] RBAC: Authorizing `identity:delete_user(user_id=99be694ce83647f9aaf519673d73dde4)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:05.204318 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5825f4da-b55a-4760-9e6e-fc99d34bc5fa admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:05.679224 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c10f4bb-cbc8-43fb-aa3e-8404ff57ab96 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:05.680458 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-1c10f4bb-cbc8-43fb-aa3e-8404ff57ab96 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:05.698545 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c10f4bb-cbc8-43fb-aa3e-8404ff57ab96 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:05.710601 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb41f64-35e6-418b-b1fe-6a1b173df6bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:05.711375 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb41f64-35e6-418b-b1fe-6a1b173df6bd admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:05.711668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb41f64-35e6-418b-b1fe-6a1b173df6bd admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:05.712040 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb41f64-35e6-418b-b1fe-6a1b173df6bd admin admin] PATH_INFO: `/v3/users/bb1038d2409e4f8588edee88a3f8d89d` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:05.713331 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb41f64-35e6-418b-b1fe-6a1b173df6bd admin admin] RBAC: Authorizing `identity:delete_user(user_id=bb1038d2409e4f8588edee88a3f8d89d)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:05.715359 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb41f64-35e6-418b-b1fe-6a1b173df6bd admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:06.287365 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-2cb41f64-35e6-418b-b1fe-6a1b173df6bd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bb1038d2409e4f8588edee88a3f8d89d'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:06.289962 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-2cb41f64-35e6-418b-b1fe-6a1b173df6bd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bb1038d2409e4f8588edee88a3f8d89d'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:06.292490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-2cb41f64-35e6-418b-b1fe-6a1b173df6bd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bb1038d2409e4f8588edee88a3f8d89d'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:06.302208 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-5825f4da-b55a-4760-9e6e-fc99d34bc5fa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '99be694ce83647f9aaf519673d73dde4'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:06.315864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-2cb41f64-35e6-418b-b1fe-6a1b173df6bd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bb1038d2409e4f8588edee88a3f8d89d'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:06.336542 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-2cb41f64-35e6-418b-b1fe-6a1b173df6bd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bb1038d2409e4f8588edee88a3f8d89d'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:06.426070 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-5825f4da-b55a-4760-9e6e-fc99d34bc5fa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '99be694ce83647f9aaf519673d73dde4'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:06.427715 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-5825f4da-b55a-4760-9e6e-fc99d34bc5fa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '99be694ce83647f9aaf519673d73dde4'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:06.437082 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-5825f4da-b55a-4760-9e6e-fc99d34bc5fa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '99be694ce83647f9aaf519673d73dde4'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:06.455501 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-5825f4da-b55a-4760-9e6e-fc99d34bc5fa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '99be694ce83647f9aaf519673d73dde4'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:06.464823 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 816/1655] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:39:05 2022] DELETE /identity/v3/users/99be694ce83647f9aaf519673d73dde4 => generated 0 bytes in 1454 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:06.466786 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-befe0654-3c77-47a1-8f8b-72857ee9c432 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:06.467447 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-befe0654-3c77-47a1-8f8b-72857ee9c432 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:06.467834 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-befe0654-3c77-47a1-8f8b-72857ee9c432 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:06.536408 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 840/1656] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:39:05 2022] DELETE /identity/v3/users/bb1038d2409e4f8588edee88a3f8d89d => generated 0 bytes in 858 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:06.540630 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc294f4-c8cc-4c0e-b203-63ed1c55cf82 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:06.540891 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc294f4-c8cc-4c0e-b203-63ed1c55cf82 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:06.541134 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc294f4-c8cc-4c0e-b203-63ed1c55cf82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:06.578761 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-befe0654-3c77-47a1-8f8b-72857ee9c432 None None] MFA Rules not processed for user `18a8a578f3554b3dbf53a50db6c94faf`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:06.581819 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-befe0654-3c77-47a1-8f8b-72857ee9c432 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:06.601676 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-3bc294f4-c8cc-4c0e-b203-63ed1c55cf82 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:06.602882 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-3bc294f4-c8cc-4c0e-b203-63ed1c55cf82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:06.756906 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 817/1657] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:06 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 292 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:06.762606 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-befe0654-3c77-47a1-8f8b-72857ee9c432 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:06.764008 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-befe0654-3c77-47a1-8f8b-72857ee9c432 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:06.764735 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 841/1658] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:06.781010 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-befe0654-3c77-47a1-8f8b-72857ee9c432 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:06.787363 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2731008-8fa6-4653-897c-1769f8cdc544 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:06.788245 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2731008-8fa6-4653-897c-1769f8cdc544 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:06.788596 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2731008-8fa6-4653-897c-1769f8cdc544 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:06.789514 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2731008-8fa6-4653-897c-1769f8cdc544 admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:06.789758 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2731008-8fa6-4653-897c-1769f8cdc544 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:06.792037 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2731008-8fa6-4653-897c-1769f8cdc544 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:06.810842 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc294f4-c8cc-4c0e-b203-63ed1c55cf82 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:06.813945 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-3bc294f4-c8cc-4c0e-b203-63ed1c55cf82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:06.929968 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 818/1659] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:39:06 2022] POST /identity/v3/projects => generated 403 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:39:06.941851 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2731008-8fa6-4653-897c-1769f8cdc544 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:06.995775 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc294f4-c8cc-4c0e-b203-63ed1c55cf82 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:07.132684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2731008-8fa6-4653-897c-1769f8cdc544 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:07.145794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d076b6-9214-4140-8016-c3295a96485d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:07.146929 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d076b6-9214-4140-8016-c3295a96485d admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:07.146929 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d076b6-9214-4140-8016-c3295a96485d admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:07.147144 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d076b6-9214-4140-8016-c3295a96485d admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:07.147836 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d076b6-9214-4140-8016-c3295a96485d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:07.150507 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d076b6-9214-4140-8016-c3295a96485d admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:07.159478 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2eee75-f12f-40e7-b178-3382b74e57ac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:07.160248 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2eee75-f12f-40e7-b178-3382b74e57ac service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:07.163747 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2eee75-f12f-40e7-b178-3382b74e57ac service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:07.163747 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2eee75-f12f-40e7-b178-3382b74e57ac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:07.179621 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2eee75-f12f-40e7-b178-3382b74e57ac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:07.181729 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2eee75-f12f-40e7-b178-3382b74e57ac service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:07.205994 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 842/1660] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:06 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 395 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:39:07.339048 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 819/1661] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:39:06 2022] POST /identity/v3/users => generated 308 bytes in 399 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:07.343182 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2eee75-f12f-40e7-b178-3382b74e57ac service neutron] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:07.361565 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2eee75-f12f-40e7-b178-3382b74e57ac service neutron] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:07.376054 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b11c40-d5af-4fb5-903b-c93ceac80afa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:07.377955 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b11c40-d5af-4fb5-903b-c93ceac80afa admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:07.378835 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b11c40-d5af-4fb5-903b-c93ceac80afa admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:07.380178 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b11c40-d5af-4fb5-903b-c93ceac80afa admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:07.381349 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6b11c40-d5af-4fb5-903b-c93ceac80afa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:07.389708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6b11c40-d5af-4fb5-903b-c93ceac80afa admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:07.417985 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 843/1662] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:07 2022] GET /identity/v3/roles => generated 1420 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:07.421687 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d076b6-9214-4140-8016-c3295a96485d admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:07.463232 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d076b6-9214-4140-8016-c3295a96485d admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:07.470115 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3a0d40-edac-46e2-9c04-94626bfc8e9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:07.472671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3a0d40-edac-46e2-9c04-94626bfc8e9a admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:07.473141 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3a0d40-edac-46e2-9c04-94626bfc8e9a admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:07.473563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3a0d40-edac-46e2-9c04-94626bfc8e9a admin admin] PATH_INFO: `/v3/projects/0a2e89491fcb49dca79518c2d50a71c7/users/b2c5221ac1a8421db778e360fe8868d8/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:07.512830 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3a0d40-edac-46e2-9c04-94626bfc8e9a admin admin] RBAC: Authorizing `identity:create_grant(project_id=0a2e89491fcb49dca79518c2d50a71c7, user_id=b2c5221ac1a8421db778e360fe8868d8, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:07.516409 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3a0d40-edac-46e2-9c04-94626bfc8e9a admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:07.660496 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 820/1663] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:39:07 2022] PUT /identity/v3/projects/0a2e89491fcb49dca79518c2d50a71c7/users/b2c5221ac1a8421db778e360fe8868d8/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:07.664507 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b11c40-d5af-4fb5-903b-c93ceac80afa admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:07.687451 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b11c40-d5af-4fb5-903b-c93ceac80afa admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:07.767691 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:07.865948 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6973700c-1bc5-4fe1-ac9b-ff7400433bcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:07.867543 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6973700c-1bc5-4fe1-ac9b-ff7400433bcd admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:07.867894 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6973700c-1bc5-4fe1-ac9b-ff7400433bcd admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:07.871587 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6973700c-1bc5-4fe1-ac9b-ff7400433bcd admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:07.873977 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6973700c-1bc5-4fe1-ac9b-ff7400433bcd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:07.878350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6973700c-1bc5-4fe1-ac9b-ff7400433bcd admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:07.903911 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 844/1664] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:07 2022] GET /identity/v3/roles => generated 1420 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:07.910698 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3a0d40-edac-46e2-9c04-94626bfc8e9a admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:07.930134 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3a0d40-edac-46e2-9c04-94626bfc8e9a admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:07.949631 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96c76937-bd82-4c39-ad2a-059b65666871 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:07.949631 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c76937-bd82-4c39-ad2a-059b65666871 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:07.949631 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c76937-bd82-4c39-ad2a-059b65666871 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:07.949631 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c76937-bd82-4c39-ad2a-059b65666871 admin admin] PATH_INFO: `/v3/projects/0a2e89491fcb49dca79518c2d50a71c7/users/b2c5221ac1a8421db778e360fe8868d8/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:07.966151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96c76937-bd82-4c39-ad2a-059b65666871 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0a2e89491fcb49dca79518c2d50a71c7, user_id=b2c5221ac1a8421db778e360fe8868d8, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:07.968340 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96c76937-bd82-4c39-ad2a-059b65666871 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:08.014183 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 821/1665] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:39:07 2022] PUT /identity/v3/projects/0a2e89491fcb49dca79518c2d50a71c7/users/b2c5221ac1a8421db778e360fe8868d8/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:39:08.170406 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6973700c-1bc5-4fe1-ac9b-ff7400433bcd admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:08.173718 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-6973700c-1bc5-4fe1-ac9b-ff7400433bcd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:08.199723 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6973700c-1bc5-4fe1-ac9b-ff7400433bcd admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:08.242464 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:08.372601 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c287d0a-99b9-41d1-9569-4da38a4e6ecc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:08.373507 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c287d0a-99b9-41d1-9569-4da38a4e6ecc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:08.373797 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c287d0a-99b9-41d1-9569-4da38a4e6ecc admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:08.374080 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c287d0a-99b9-41d1-9569-4da38a4e6ecc admin admin] PATH_INFO: `/v3/projects/e9d33678c4554583bfa489acbfeebc29` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:08.377326 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c287d0a-99b9-41d1-9569-4da38a4e6ecc admin admin] RBAC: Authorizing `identity:delete_project(project_id=e9d33678c4554583bfa489acbfeebc29)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:08.382778 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c287d0a-99b9-41d1-9569-4da38a4e6ecc admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:08.643743 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-7c287d0a-99b9-41d1-9569-4da38a4e6ecc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e9d33678c4554583bfa489acbfeebc29', 'request_id': 'req-7c287d0a-99b9-41d1-9569-4da38a4e6ecc'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:08.659314 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-7c287d0a-99b9-41d1-9569-4da38a4e6ecc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e9d33678c4554583bfa489acbfeebc29', 'request_id': 'req-7c287d0a-99b9-41d1-9569-4da38a4e6ecc'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:08.779577 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-7c287d0a-99b9-41d1-9569-4da38a4e6ecc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e9d33678c4554583bfa489acbfeebc29', 'request_id': 'req-7c287d0a-99b9-41d1-9569-4da38a4e6ecc'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:08.807297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-7c287d0a-99b9-41d1-9569-4da38a4e6ecc admin admin] The token cache is being invalidate because project e9d33678c4554583bfa489acbfeebc29 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:39:08.807801 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-7c287d0a-99b9-41d1-9569-4da38a4e6ecc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:08.810098 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-7c287d0a-99b9-41d1-9569-4da38a4e6ecc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:08.816376 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 845/1666] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:39:08 2022] DELETE /identity/v3/projects/e9d33678c4554583bfa489acbfeebc29 => generated 0 bytes in 646 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:08.825694 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35f328f-2f7e-4b54-a9eb-8cda3abbf1df None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:08.826243 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35f328f-2f7e-4b54-a9eb-8cda3abbf1df None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:08.827677 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35f328f-2f7e-4b54-a9eb-8cda3abbf1df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:08.878464 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-b35f328f-2f7e-4b54-a9eb-8cda3abbf1df None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:08.881276 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-b35f328f-2f7e-4b54-a9eb-8cda3abbf1df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:09.072488 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 822/1667] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 252 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:09.081641 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4d3a1a-74b2-4e6c-bed5-fd9561bea510 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:09.082173 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4d3a1a-74b2-4e6c-bed5-fd9561bea510 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:09.082599 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4d3a1a-74b2-4e6c-bed5-fd9561bea510 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:09.120433 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-1e4d3a1a-74b2-4e6c-bed5-fd9561bea510 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:09.123163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-1e4d3a1a-74b2-4e6c-bed5-fd9561bea510 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:09.166926 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 846/1668] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:09.171308 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35f328f-2f7e-4b54-a9eb-8cda3abbf1df None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:09.185319 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35f328f-2f7e-4b54-a9eb-8cda3abbf1df None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:09.192778 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86400481-0256-48f1-8031-cb8c58226bb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:09.193995 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86400481-0256-48f1-8031-cb8c58226bb2 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:09.194970 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86400481-0256-48f1-8031-cb8c58226bb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:09.195471 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86400481-0256-48f1-8031-cb8c58226bb2 admin admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:09.197153 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86400481-0256-48f1-8031-cb8c58226bb2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:39:09.197691 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86400481-0256-48f1-8031-cb8c58226bb2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:09.200732 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86400481-0256-48f1-8031-cb8c58226bb2 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:09.226330 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 823/1669] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:39:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:39:09.231182 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4d3a1a-74b2-4e6c-bed5-fd9561bea510 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:09.259116 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4d3a1a-74b2-4e6c-bed5-fd9561bea510 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:09.276598 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26fecf6-46ce-45a7-bba6-9ff4243366e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:09.277851 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26fecf6-46ce-45a7-bba6-9ff4243366e8 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:09.279467 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26fecf6-46ce-45a7-bba6-9ff4243366e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:09.279872 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26fecf6-46ce-45a7-bba6-9ff4243366e8 admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:09.282569 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26fecf6-46ce-45a7-bba6-9ff4243366e8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:09.287810 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26fecf6-46ce-45a7-bba6-9ff4243366e8 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:09.346766 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86400481-0256-48f1-8031-cb8c58226bb2 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:09.349093 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-86400481-0256-48f1-8031-cb8c58226bb2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:09.366968 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86400481-0256-48f1-8031-cb8c58226bb2 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:09.374456 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691af65c-a8a2-4d73-b883-2aeaea9f0b8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:09.377812 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691af65c-a8a2-4d73-b883-2aeaea9f0b8d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:09.378218 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691af65c-a8a2-4d73-b883-2aeaea9f0b8d admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:09.378638 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691af65c-a8a2-4d73-b883-2aeaea9f0b8d admin admin] PATH_INFO: `/v3/projects/291ab35011594a95b495a19d62d15077` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:09.380981 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691af65c-a8a2-4d73-b883-2aeaea9f0b8d admin admin] RBAC: Authorizing `identity:delete_project(project_id=291ab35011594a95b495a19d62d15077)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:09.385480 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691af65c-a8a2-4d73-b883-2aeaea9f0b8d admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:09.408998 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 847/1670] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:39:09 2022] POST /identity/v3/projects => generated 381 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:09.414307 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26fecf6-46ce-45a7-bba6-9ff4243366e8 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:09.589655 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-691af65c-a8a2-4d73-b883-2aeaea9f0b8d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '291ab35011594a95b495a19d62d15077', 'request_id': 'req-691af65c-a8a2-4d73-b883-2aeaea9f0b8d'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:09.601135 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26fecf6-46ce-45a7-bba6-9ff4243366e8 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:09.612147 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-691af65c-a8a2-4d73-b883-2aeaea9f0b8d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '291ab35011594a95b495a19d62d15077', 'request_id': 'req-691af65c-a8a2-4d73-b883-2aeaea9f0b8d'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:09.629660 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-691af65c-a8a2-4d73-b883-2aeaea9f0b8d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '291ab35011594a95b495a19d62d15077', 'request_id': 'req-691af65c-a8a2-4d73-b883-2aeaea9f0b8d'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:09.724240 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-691af65c-a8a2-4d73-b883-2aeaea9f0b8d admin admin] The token cache is being invalidate because project 291ab35011594a95b495a19d62d15077 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:39:09.724885 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-691af65c-a8a2-4d73-b883-2aeaea9f0b8d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:09.726415 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-691af65c-a8a2-4d73-b883-2aeaea9f0b8d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:09.733016 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 824/1671] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:39:09 2022] DELETE /identity/v3/projects/291ab35011594a95b495a19d62d15077 => generated 0 bytes in 386 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:09.739433 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d732f460-7049-4471-b694-55c9524bf31e None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:09.739734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d732f460-7049-4471-b694-55c9524bf31e None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:09.740017 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d732f460-7049-4471-b694-55c9524bf31e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:09.752020 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849bdfed-3e01-4944-acea-ac34855d355b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:09.752721 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849bdfed-3e01-4944-acea-ac34855d355b admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:09.753012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849bdfed-3e01-4944-acea-ac34855d355b admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:09.753306 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849bdfed-3e01-4944-acea-ac34855d355b admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:09.754008 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-849bdfed-3e01-4944-acea-ac34855d355b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:09.755986 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-849bdfed-3e01-4944-acea-ac34855d355b admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:09.784180 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-d732f460-7049-4471-b694-55c9524bf31e None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:09.786584 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-d732f460-7049-4471-b694-55c9524bf31e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:09.799089 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 825/1672] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 61 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:09.808887 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc12df3c-d117-4b69-804d-6c87a6d269ed None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:09.814036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc12df3c-d117-4b69-804d-6c87a6d269ed None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:09.814036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc12df3c-d117-4b69-804d-6c87a6d269ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:09.839037 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-cc12df3c-d117-4b69-804d-6c87a6d269ed None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:09.841840 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-cc12df3c-d117-4b69-804d-6c87a6d269ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:09.859237 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 826/1673] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:09.860467 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc12df3c-d117-4b69-804d-6c87a6d269ed None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:09.880351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc12df3c-d117-4b69-804d-6c87a6d269ed None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:09.890486 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8fbde0-6319-4117-80a1-fbc674b63a71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:09.892245 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8fbde0-6319-4117-80a1-fbc674b63a71 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:09.903455 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8fbde0-6319-4117-80a1-fbc674b63a71 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:09.903455 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8fbde0-6319-4117-80a1-fbc674b63a71 admin admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:09.903455 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8fbde0-6319-4117-80a1-fbc674b63a71 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:39:09.903455 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8fbde0-6319-4117-80a1-fbc674b63a71 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:09.903455 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8fbde0-6319-4117-80a1-fbc674b63a71 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:09.903958 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 848/1674] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:39:09 2022] POST /identity/v3/users => generated 291 bytes in 487 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:09.905944 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849bdfed-3e01-4944-acea-ac34855d355b admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:09.914756 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-849bdfed-3e01-4944-acea-ac34855d355b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:09.916543 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 827/1675] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:39:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:39:09.923239 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8fbde0-6319-4117-80a1-fbc674b63a71 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:09.939204 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8fbde0-6319-4117-80a1-fbc674b63a71 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:09.945480 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77641237-26b0-429e-9352-f30b547b3332 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:09.946580 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77641237-26b0-429e-9352-f30b547b3332 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:09.946881 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77641237-26b0-429e-9352-f30b547b3332 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:09.947087 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849bdfed-3e01-4944-acea-ac34855d355b admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:09.947215 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77641237-26b0-429e-9352-f30b547b3332 admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:09.947992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77641237-26b0-429e-9352-f30b547b3332 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:09.951980 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77641237-26b0-429e-9352-f30b547b3332 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:09.965995 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54bfcdc-f8e7-42f2-8af4-2480e76747a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:09.966725 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54bfcdc-f8e7-42f2-8af4-2480e76747a7 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:09.967018 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54bfcdc-f8e7-42f2-8af4-2480e76747a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:09.967300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54bfcdc-f8e7-42f2-8af4-2480e76747a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:09.968099 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b54bfcdc-f8e7-42f2-8af4-2480e76747a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:09.969792 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b54bfcdc-f8e7-42f2-8af4-2480e76747a7 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:09.995388 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 849/1676] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:09 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:09.999280 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54bfcdc-f8e7-42f2-8af4-2480e76747a7 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:10.020100 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54bfcdc-f8e7-42f2-8af4-2480e76747a7 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:10.031936 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48aaa3f6-c485-408d-92b3-ee0c726d96af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:10.033563 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48aaa3f6-c485-408d-92b3-ee0c726d96af admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:10.033869 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48aaa3f6-c485-408d-92b3-ee0c726d96af admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:10.034222 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48aaa3f6-c485-408d-92b3-ee0c726d96af admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:10.035180 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48aaa3f6-c485-408d-92b3-ee0c726d96af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:10.036980 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48aaa3f6-c485-408d-92b3-ee0c726d96af admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:10.062175 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 850/1677] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:09 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:10.067130 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48aaa3f6-c485-408d-92b3-ee0c726d96af admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:10.092297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48aaa3f6-c485-408d-92b3-ee0c726d96af admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:10.101937 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43241945-a904-48f9-803b-77795e0193f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:10.103009 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43241945-a904-48f9-803b-77795e0193f8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:10.103315 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43241945-a904-48f9-803b-77795e0193f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:10.103589 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43241945-a904-48f9-803b-77795e0193f8 admin admin] PATH_INFO: `/v3/projects/1a54f9fed45146cbaa39ecd1f6df41cc/users/f18be44ed7f64d4fae92dc6d79a84dc6/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:10.130649 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43241945-a904-48f9-803b-77795e0193f8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1a54f9fed45146cbaa39ecd1f6df41cc, user_id=f18be44ed7f64d4fae92dc6d79a84dc6, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:10.132939 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43241945-a904-48f9-803b-77795e0193f8 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:10.149187 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 828/1678] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:39:09 2022] POST /identity/v3/projects => generated 363 bytes in 228 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:10.151224 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77641237-26b0-429e-9352-f30b547b3332 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:10.188208 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=92,keystone:DELETE=2,keystone:INSERT=6 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:10.241545 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 851/1679] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:39:10 2022] PUT /identity/v3/projects/1a54f9fed45146cbaa39ecd1f6df41cc/users/f18be44ed7f64d4fae92dc6d79a84dc6/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:10.252930 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43241945-a904-48f9-803b-77795e0193f8 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:10.258181 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-43241945-a904-48f9-803b-77795e0193f8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:10.345546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77641237-26b0-429e-9352-f30b547b3332 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:10.405079 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43241945-a904-48f9-803b-77795e0193f8 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:10.418803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a6fcf5-513a-4544-bf33-12f565aa0f15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:10.419782 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a6fcf5-513a-4544-bf33-12f565aa0f15 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:10.420134 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a6fcf5-513a-4544-bf33-12f565aa0f15 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:10.420507 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a6fcf5-513a-4544-bf33-12f565aa0f15 admin admin] PATH_INFO: `/v3/users/4d8217a2573048819d35ac58c5c306a2` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:10.425009 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0a6fcf5-513a-4544-bf33-12f565aa0f15 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4d8217a2573048819d35ac58c5c306a2)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:10.430240 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0a6fcf5-513a-4544-bf33-12f565aa0f15 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:10.445556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:10.527590 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20d2b74-0d55-43e2-86d1-74e563d707d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:10.528962 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20d2b74-0d55-43e2-86d1-74e563d707d2 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:10.529426 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20d2b74-0d55-43e2-86d1-74e563d707d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:10.529936 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20d2b74-0d55-43e2-86d1-74e563d707d2 admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:10.533623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e20d2b74-0d55-43e2-86d1-74e563d707d2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:10.535973 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e20d2b74-0d55-43e2-86d1-74e563d707d2 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:10.678238 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 829/1680] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:39:10 2022] POST /identity/v3/users => generated 282 bytes in 528 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:10.692827 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f34b0e-a646-4bf2-a9c2-bc8ec94c8034 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:10.698038 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f34b0e-a646-4bf2-a9c2-bc8ec94c8034 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:10.698038 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f34b0e-a646-4bf2-a9c2-bc8ec94c8034 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:10.742920 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-a0a6fcf5-513a-4544-bf33-12f565aa0f15 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4d8217a2573048819d35ac58c5c306a2'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:10.744618 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-b7f34b0e-a646-4bf2-a9c2-bc8ec94c8034 None None] MFA Rules not processed for user `f18be44ed7f64d4fae92dc6d79a84dc6`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:10.745710 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-a0a6fcf5-513a-4544-bf33-12f565aa0f15 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4d8217a2573048819d35ac58c5c306a2'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:10.746107 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-b7f34b0e-a646-4bf2-a9c2-bc8ec94c8034 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:10.746934 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-a0a6fcf5-513a-4544-bf33-12f565aa0f15 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4d8217a2573048819d35ac58c5c306a2'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:10.756942 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-a0a6fcf5-513a-4544-bf33-12f565aa0f15 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4d8217a2573048819d35ac58c5c306a2'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:10.812124 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-a0a6fcf5-513a-4544-bf33-12f565aa0f15 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4d8217a2573048819d35ac58c5c306a2'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:10.919720 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 852/1681] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:39:10 2022] DELETE /identity/v3/users/4d8217a2573048819d35ac58c5c306a2 => generated 0 bytes in 677 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:10.933358 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a6fcf5-513a-4544-bf33-12f565aa0f15 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:10.933358 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-a0a6fcf5-513a-4544-bf33-12f565aa0f15 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:10.945235 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 830/1682] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:10 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 253 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:10.954748 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4be062-caf6-4af8-8ef6-9a76673aa01f None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:10.959205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4be062-caf6-4af8-8ef6-9a76673aa01f None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:10.959532 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4be062-caf6-4af8-8ef6-9a76673aa01f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:11.007627 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-8b4be062-caf6-4af8-8ef6-9a76673aa01f None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:11.012258 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-8b4be062-caf6-4af8-8ef6-9a76673aa01f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:11.174838 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 831/1683] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 229 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:11.176309 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4be062-caf6-4af8-8ef6-9a76673aa01f None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:11.181203 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a6fcf5-513a-4544-bf33-12f565aa0f15 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:11.181724 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-8b4be062-caf6-4af8-8ef6-9a76673aa01f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:11.208398 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96ca9f2-a61f-4b52-b42f-7397f6aa9369 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:11.209213 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96ca9f2-a61f-4b52-b42f-7397f6aa9369 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:11.209571 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96ca9f2-a61f-4b52-b42f-7397f6aa9369 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:11.209928 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96ca9f2-a61f-4b52-b42f-7397f6aa9369 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:11.210787 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d96ca9f2-a61f-4b52-b42f-7397f6aa9369 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:11.214351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4be062-caf6-4af8-8ef6-9a76673aa01f None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:11.217235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d96ca9f2-a61f-4b52-b42f-7397f6aa9369 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:11.236564 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4713932-ff7a-45d0-bfd9-70ce2c127cb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:11.237257 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4713932-ff7a-45d0-bfd9-70ce2c127cb1 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:11.237542 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4713932-ff7a-45d0-bfd9-70ce2c127cb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:11.237979 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4713932-ff7a-45d0-bfd9-70ce2c127cb1 admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:11.238750 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4713932-ff7a-45d0-bfd9-70ce2c127cb1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:11.240742 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4713932-ff7a-45d0-bfd9-70ce2c127cb1 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:11.259024 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 853/1684] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:10 2022] GET /identity/v3/roles => generated 1420 bytes in 336 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:11.259024 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96ca9f2-a61f-4b52-b42f-7397f6aa9369 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:11.265961 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-d96ca9f2-a61f-4b52-b42f-7397f6aa9369 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:11.410675 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 832/1685] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:39:11 2022] POST /identity/v3/projects => generated 377 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:11.411874 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4713932-ff7a-45d0-bfd9-70ce2c127cb1 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:11.412400 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96ca9f2-a61f-4b52-b42f-7397f6aa9369 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:11.576835 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4713932-ff7a-45d0-bfd9-70ce2c127cb1 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:11.587323 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd38fbd5-eb18-46fa-9509-bfa41b0efd3d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:11.588034 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd38fbd5-eb18-46fa-9509-bfa41b0efd3d service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:11.588862 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd38fbd5-eb18-46fa-9509-bfa41b0efd3d service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:11.589155 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd38fbd5-eb18-46fa-9509-bfa41b0efd3d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:11.592209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27156c3e-2984-4fba-87aa-2a326d6c11da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:11.593502 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27156c3e-2984-4fba-87aa-2a326d6c11da admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:11.595384 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27156c3e-2984-4fba-87aa-2a326d6c11da admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:11.595760 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27156c3e-2984-4fba-87aa-2a326d6c11da admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:11.596592 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27156c3e-2984-4fba-87aa-2a326d6c11da admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:11.598909 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27156c3e-2984-4fba-87aa-2a326d6c11da admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:11.614075 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd38fbd5-eb18-46fa-9509-bfa41b0efd3d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:11.616606 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd38fbd5-eb18-46fa-9509-bfa41b0efd3d service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:11.631631 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 833/1686] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:11 2022] GET /identity/v3/roles => generated 1420 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:11.636016 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27156c3e-2984-4fba-87aa-2a326d6c11da admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:11.646780 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 854/1687] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:11 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 390 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:39:11.648487 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd38fbd5-eb18-46fa-9509-bfa41b0efd3d service neutron] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:11.659237 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27156c3e-2984-4fba-87aa-2a326d6c11da admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:11.668985 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd38fbd5-eb18-46fa-9509-bfa41b0efd3d service neutron] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:11.672275 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a33e753-93f6-4e95-a6b3-4e73f8a96c48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:11.673514 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a33e753-93f6-4e95-a6b3-4e73f8a96c48 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:11.673832 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a33e753-93f6-4e95-a6b3-4e73f8a96c48 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:11.674542 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a33e753-93f6-4e95-a6b3-4e73f8a96c48 admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:11.675465 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a33e753-93f6-4e95-a6b3-4e73f8a96c48 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:11.678229 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a33e753-93f6-4e95-a6b3-4e73f8a96c48 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:11.686677 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c50d65-cbfd-46dd-9ded-2632e3159e21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:11.687946 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c50d65-cbfd-46dd-9ded-2632e3159e21 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:11.688077 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c50d65-cbfd-46dd-9ded-2632e3159e21 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:11.688951 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c50d65-cbfd-46dd-9ded-2632e3159e21 admin admin] PATH_INFO: `/v3/projects/0d9858aaa86e461a9e13a9521a562c2b/users/aad7479f671a4d0da933c72688d867ba/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:11.719028 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c50d65-cbfd-46dd-9ded-2632e3159e21 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0d9858aaa86e461a9e13a9521a562c2b, user_id=aad7479f671a4d0da933c72688d867ba, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:11.723119 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c50d65-cbfd-46dd-9ded-2632e3159e21 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:11.777866 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=93,keystone:INSERT=6,keystone:DELETE=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:11.818866 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 855/1688] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:39:11 2022] PUT /identity/v3/projects/0d9858aaa86e461a9e13a9521a562c2b/users/aad7479f671a4d0da933c72688d867ba/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:11.819314 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 834/1689] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:39:11 2022] POST /identity/v3/users => generated 295 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:11.830847 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3bdb7b-27fe-4bef-bd4e-8a869348925c None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:11.830847 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3bdb7b-27fe-4bef-bd4e-8a869348925c None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:11.830847 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3bdb7b-27fe-4bef-bd4e-8a869348925c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:11.832250 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a33e753-93f6-4e95-a6b3-4e73f8a96c48 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:11.860581 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a33e753-93f6-4e95-a6b3-4e73f8a96c48 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:11.934387 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-cb3bdb7b-27fe-4bef-bd4e-8a869348925c None None] MFA Rules not processed for user `aad7479f671a4d0da933c72688d867ba`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:11.947780 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-cb3bdb7b-27fe-4bef-bd4e-8a869348925c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:12.071598 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307018a9-ddc9-4338-ba35-93d0de523cc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:12.072931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307018a9-ddc9-4338-ba35-93d0de523cc8 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:12.073352 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307018a9-ddc9-4338-ba35-93d0de523cc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:12.073771 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307018a9-ddc9-4338-ba35-93d0de523cc8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:12.074801 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-307018a9-ddc9-4338-ba35-93d0de523cc8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:12.076559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-307018a9-ddc9-4338-ba35-93d0de523cc8 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:12.106113 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 835/1690] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:11 2022] GET /identity/v3/roles => generated 1420 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:12.107312 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307018a9-ddc9-4338-ba35-93d0de523cc8 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:12.123621 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307018a9-ddc9-4338-ba35-93d0de523cc8 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:12.139084 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93314197-739c-470c-8f03-a3ec19f9e1a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:12.140425 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93314197-739c-470c-8f03-a3ec19f9e1a1 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:12.140859 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93314197-739c-470c-8f03-a3ec19f9e1a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:12.141834 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93314197-739c-470c-8f03-a3ec19f9e1a1 admin admin] PATH_INFO: `/v3/projects/0b2661b3862b49629d3ef4002215cd49/users/4eb3ab323723400891c7088333bb56e9/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:12.149500 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 856/1691] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:11 2022] POST /identity/v3/auth/tokens => generated 3230 bytes in 321 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:12.153665 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3bdb7b-27fe-4bef-bd4e-8a869348925c None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:12.179508 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:12.180025 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3bdb7b-27fe-4bef-bd4e-8a869348925c None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:12.180794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93314197-739c-470c-8f03-a3ec19f9e1a1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0b2661b3862b49629d3ef4002215cd49, user_id=4eb3ab323723400891c7088333bb56e9, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:12.184168 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93314197-739c-470c-8f03-a3ec19f9e1a1 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:12.191556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f76858-e383-41cc-a5e3-265d3e470a6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:12.192408 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f76858-e383-41cc-a5e3-265d3e470a6d admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:12.192757 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f76858-e383-41cc-a5e3-265d3e470a6d admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:12.193499 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f76858-e383-41cc-a5e3-265d3e470a6d admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:12.197007 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69f76858-e383-41cc-a5e3-265d3e470a6d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:12.202030 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69f76858-e383-41cc-a5e3-265d3e470a6d admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:12.311208 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 836/1692] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:39:12 2022] PUT /identity/v3/projects/0b2661b3862b49629d3ef4002215cd49/users/4eb3ab323723400891c7088333bb56e9/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:12.322629 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93314197-739c-470c-8f03-a3ec19f9e1a1 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:12.356396 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 857/1693] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:39:12 2022] POST /identity/v3/projects => generated 365 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:12.360952 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f76858-e383-41cc-a5e3-265d3e470a6d admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:12.522000 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93314197-739c-470c-8f03-a3ec19f9e1a1 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:12.579081 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f76858-e383-41cc-a5e3-265d3e470a6d admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:12.593118 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d89fb4-9af8-454b-8cf8-4cb4749a7c67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:12.595936 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d89fb4-9af8-454b-8cf8-4cb4749a7c67 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:12.598182 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d89fb4-9af8-454b-8cf8-4cb4749a7c67 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:12.600162 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d89fb4-9af8-454b-8cf8-4cb4749a7c67 admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:12.600957 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d89fb4-9af8-454b-8cf8-4cb4749a7c67 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:12.613844 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d89fb4-9af8-454b-8cf8-4cb4749a7c67 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:12.707915 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519bec48-3182-4485-8ea6-f5495d93e54a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:12.710941 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519bec48-3182-4485-8ea6-f5495d93e54a admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:12.711525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519bec48-3182-4485-8ea6-f5495d93e54a admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:12.711949 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519bec48-3182-4485-8ea6-f5495d93e54a admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:12.712913 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519bec48-3182-4485-8ea6-f5495d93e54a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:12.715188 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519bec48-3182-4485-8ea6-f5495d93e54a admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:12.762215 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 837/1694] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:12 2022] GET /identity/v3/roles => generated 1420 bytes in 446 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:12.765216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 858/1695] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:39:12 2022] POST /identity/v3/users => generated 289 bytes in 406 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:12.767834 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d89fb4-9af8-454b-8cf8-4cb4749a7c67 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:12.768337 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519bec48-3182-4485-8ea6-f5495d93e54a admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:12.786729 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519bec48-3182-4485-8ea6-f5495d93e54a admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:12.791653 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d89fb4-9af8-454b-8cf8-4cb4749a7c67 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:12.794745 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6be603e-7999-4b8c-95f3-065aeadc270c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:12.795697 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6be603e-7999-4b8c-95f3-065aeadc270c admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:12.798175 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6be603e-7999-4b8c-95f3-065aeadc270c admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:12.799476 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b814d5b-30a1-4dc4-89a2-14944357aec2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:12.801454 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6be603e-7999-4b8c-95f3-065aeadc270c admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:12.802116 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6be603e-7999-4b8c-95f3-065aeadc270c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:12.803069 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b814d5b-30a1-4dc4-89a2-14944357aec2 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:12.803550 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b814d5b-30a1-4dc4-89a2-14944357aec2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:12.804022 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b814d5b-30a1-4dc4-89a2-14944357aec2 admin admin] PATH_INFO: `/v3/projects/0b2661b3862b49629d3ef4002215cd49/users/4eb3ab323723400891c7088333bb56e9/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:12.805977 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6be603e-7999-4b8c-95f3-065aeadc270c admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:12.807478 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b814d5b-30a1-4dc4-89a2-14944357aec2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0b2661b3862b49629d3ef4002215cd49, user_id=4eb3ab323723400891c7088333bb56e9, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:12.810758 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b814d5b-30a1-4dc4-89a2-14944357aec2 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:12.830568 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 859/1696] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:39:12 2022] PUT /identity/v3/projects/0b2661b3862b49629d3ef4002215cd49/users/4eb3ab323723400891c7088333bb56e9/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:12.834804 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 838/1697] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:12 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:12.838243 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b814d5b-30a1-4dc4-89a2-14944357aec2 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:12.840915 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73c874c-626e-45e0-8df5-6671422a4b23 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:12.841254 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73c874c-626e-45e0-8df5-6671422a4b23 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:12.841543 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73c874c-626e-45e0-8df5-6671422a4b23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:12.876445 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-d73c874c-626e-45e0-8df5-6671422a4b23 None None] MFA Rules not processed for user `4eb3ab323723400891c7088333bb56e9`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:12.878180 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-d73c874c-626e-45e0-8df5-6671422a4b23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:12.948417 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b814d5b-30a1-4dc4-89a2-14944357aec2 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:12.960725 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f98f900-f47f-4ce1-97be-6f565a8b3fed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:12.961723 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f98f900-f47f-4ce1-97be-6f565a8b3fed admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:12.962014 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f98f900-f47f-4ce1-97be-6f565a8b3fed admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:12.962296 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f98f900-f47f-4ce1-97be-6f565a8b3fed admin admin] PATH_INFO: `/v3/projects/b6fd3357973949308ec8a6ffb78dd938/users/a48a3de2cdec4911bc802036e5393302/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:13.021694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f98f900-f47f-4ce1-97be-6f565a8b3fed admin admin] RBAC: Authorizing `identity:create_grant(project_id=b6fd3357973949308ec8a6ffb78dd938, user_id=a48a3de2cdec4911bc802036e5393302, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:13.024564 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f98f900-f47f-4ce1-97be-6f565a8b3fed admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:13.034828 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 839/1698] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:12 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:13.064605 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73c874c-626e-45e0-8df5-6671422a4b23 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:13.066263 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-d73c874c-626e-45e0-8df5-6671422a4b23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:13.175655 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 860/1699] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:39:12 2022] PUT /identity/v3/projects/b6fd3357973949308ec8a6ffb78dd938/users/a48a3de2cdec4911bc802036e5393302/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 338 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:13.178375 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9b1da8-2848-4180-9496-739a41850b9e None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:13.178375 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9b1da8-2848-4180-9496-739a41850b9e None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:13.178375 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd9b1da8-2848-4180-9496-739a41850b9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:13.316211 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73c874c-626e-45e0-8df5-6671422a4b23 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:13.327833 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-fd9b1da8-2848-4180-9496-739a41850b9e None None] MFA Rules not processed for user `b2c5221ac1a8421db778e360fe8868d8`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:13.336648 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-fd9b1da8-2848-4180-9496-739a41850b9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:13.390355 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d457ff3-e6d6-463d-bc0a-c058f1764267 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:13.392068 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d457ff3-e6d6-463d-bc0a-c058f1764267 service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:13.392872 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d457ff3-e6d6-463d-bc0a-c058f1764267 service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:13.393394 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d457ff3-e6d6-463d-bc0a-c058f1764267 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:13.515036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d457ff3-e6d6-463d-bc0a-c058f1764267 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:13.517831 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d457ff3-e6d6-463d-bc0a-c058f1764267 service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:13.530503 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 840/1700] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:13 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 467 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:39:13.531989 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d457ff3-e6d6-463d-bc0a-c058f1764267 service cinder] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:13.616738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d457ff3-e6d6-463d-bc0a-c058f1764267 service cinder] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:13.621634 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 861/1701] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:13 2022] POST /identity/v3/auth/tokens => generated 3337 bytes in 444 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:13.648307 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1bef3a9-5315-4968-a28c-074f68d90a65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:13.650215 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bef3a9-5315-4968-a28c-074f68d90a65 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:13.650738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bef3a9-5315-4968-a28c-074f68d90a65 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:13.654926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1bef3a9-5315-4968-a28c-074f68d90a65 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:13.655707 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1bef3a9-5315-4968-a28c-074f68d90a65 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:13.660584 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1bef3a9-5315-4968-a28c-074f68d90a65 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:13.697056 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 841/1702] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:13 2022] GET /identity/v3/roles => generated 1420 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:13.699088 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9b1da8-2848-4180-9496-739a41850b9e None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:13.723899 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd9b1da8-2848-4180-9496-739a41850b9e None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:13.755525 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4641538-0ece-4b81-8e6c-ab2d9b3fc32d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:13.757195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4641538-0ece-4b81-8e6c-ab2d9b3fc32d admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:13.757489 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4641538-0ece-4b81-8e6c-ab2d9b3fc32d admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:13.759112 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4641538-0ece-4b81-8e6c-ab2d9b3fc32d admin admin] PATH_INFO: `/v3/projects/b6fd3357973949308ec8a6ffb78dd938/users/a48a3de2cdec4911bc802036e5393302/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:13.767626 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4641538-0ece-4b81-8e6c-ab2d9b3fc32d admin admin] RBAC: Authorizing `identity:create_grant(project_id=b6fd3357973949308ec8a6ffb78dd938, user_id=a48a3de2cdec4911bc802036e5393302, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:13.783143 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4641538-0ece-4b81-8e6c-ab2d9b3fc32d admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:13.816892 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 862/1703] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:39:13 2022] PUT /identity/v3/projects/b6fd3357973949308ec8a6ffb78dd938/users/a48a3de2cdec4911bc802036e5393302/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:13.829324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4c0d7b-fc5a-4ba2-975e-2deaec3dec88 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:13.829965 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4c0d7b-fc5a-4ba2-975e-2deaec3dec88 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:13.830514 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4c0d7b-fc5a-4ba2-975e-2deaec3dec88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:13.871596 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8988bf-c5a4-4c36-b13d-ec5f100d279f None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:13.872055 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8988bf-c5a4-4c36-b13d-ec5f100d279f None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:13.872494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8988bf-c5a4-4c36-b13d-ec5f100d279f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:13.886035 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-7e4c0d7b-fc5a-4ba2-975e-2deaec3dec88 None None] MFA Rules not processed for user `a48a3de2cdec4911bc802036e5393302`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:13.887227 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-7e4c0d7b-fc5a-4ba2-975e-2deaec3dec88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:13.930648 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-9c8988bf-c5a4-4c36-b13d-ec5f100d279f None None] MFA Rules not processed for user `ce09df98c929416e8ca67b0728349bfb`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:13.932041 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-9c8988bf-c5a4-4c36-b13d-ec5f100d279f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:14.037757 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 842/1704] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:13 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:14.061049 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4c0d7b-fc5a-4ba2-975e-2deaec3dec88 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:14.063059 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-7e4c0d7b-fc5a-4ba2-975e-2deaec3dec88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:14.064213 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 863/1705] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:13 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:14.085657 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0151c4-1e3f-49e0-bd51-63b0f8fd5ded None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:14.086303 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0151c4-1e3f-49e0-bd51-63b0f8fd5ded None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:14.086838 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0151c4-1e3f-49e0-bd51-63b0f8fd5ded None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:14.165443 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:14.202269 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-8c0151c4-1e3f-49e0-bd51-63b0f8fd5ded None None] MFA Rules not processed for user `ce09df98c929416e8ca67b0728349bfb`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:14.206161 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-8c0151c4-1e3f-49e0-bd51-63b0f8fd5ded None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:14.221710 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 864/1706] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:14 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:39:14.242138 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0151c4-1e3f-49e0-bd51-63b0f8fd5ded None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:14.251454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-8c0151c4-1e3f-49e0-bd51-63b0f8fd5ded None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:14.284258 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4c0d7b-fc5a-4ba2-975e-2deaec3dec88 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:14.289905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741c41bd-caa9-4953-b4a1-c4d21e9f5e9e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:14.291098 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741c41bd-caa9-4953-b4a1-c4d21e9f5e9e service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:14.291601 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741c41bd-caa9-4953-b4a1-c4d21e9f5e9e service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:14.292116 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741c41bd-caa9-4953-b4a1-c4d21e9f5e9e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:14.325273 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-741c41bd-caa9-4953-b4a1-c4d21e9f5e9e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:14.327725 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-741c41bd-caa9-4953-b4a1-c4d21e9f5e9e service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:14.361257 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 843/1707] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:14 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:39:14.381074 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741c41bd-caa9-4953-b4a1-c4d21e9f5e9e service cinder] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:14.386590 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-741c41bd-caa9-4953-b4a1-c4d21e9f5e9e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:14.418463 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:14.516502 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0151c4-1e3f-49e0-bd51-63b0f8fd5ded None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:14.542409 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-272842f4-7c2b-4c22-aacb-86ce32eae4f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:14.544528 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272842f4-7c2b-4c22-aacb-86ce32eae4f8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:14.548246 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272842f4-7c2b-4c22-aacb-86ce32eae4f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:14.548761 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-272842f4-7c2b-4c22-aacb-86ce32eae4f8 admin admin] PATH_INFO: `/v3/projects/d0efdd0349e941ef913b9260e31d2d09` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:14.552419 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-272842f4-7c2b-4c22-aacb-86ce32eae4f8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d0efdd0349e941ef913b9260e31d2d09)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:14.557274 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-272842f4-7c2b-4c22-aacb-86ce32eae4f8 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:14.616819 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741c41bd-caa9-4953-b4a1-c4d21e9f5e9e service cinder] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:14.645406 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cefff977-51f7-4708-922c-6b5ce2de4ba2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:14.646108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefff977-51f7-4708-922c-6b5ce2de4ba2 service nova] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:14.650613 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefff977-51f7-4708-922c-6b5ce2de4ba2 service nova] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:14.651025 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefff977-51f7-4708-922c-6b5ce2de4ba2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:14.683131 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cefff977-51f7-4708-922c-6b5ce2de4ba2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:14.686080 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cefff977-51f7-4708-922c-6b5ce2de4ba2 service nova] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:14.727138 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 844/1708] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:14 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:39:14.727138 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cefff977-51f7-4708-922c-6b5ce2de4ba2 service nova] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:14.773359 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cefff977-51f7-4708-922c-6b5ce2de4ba2 service nova] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:14.796141 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-272842f4-7c2b-4c22-aacb-86ce32eae4f8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd0efdd0349e941ef913b9260e31d2d09', 'request_id': 'req-272842f4-7c2b-4c22-aacb-86ce32eae4f8'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:14.827329 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-272842f4-7c2b-4c22-aacb-86ce32eae4f8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd0efdd0349e941ef913b9260e31d2d09', 'request_id': 'req-272842f4-7c2b-4c22-aacb-86ce32eae4f8'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:14.897390 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a378751-44b8-407e-a877-cf8e7201a07d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:14.898935 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a378751-44b8-407e-a877-cf8e7201a07d service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:14.899493 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a378751-44b8-407e-a877-cf8e7201a07d service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:14.899987 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a378751-44b8-407e-a877-cf8e7201a07d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:14.939402 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-272842f4-7c2b-4c22-aacb-86ce32eae4f8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd0efdd0349e941ef913b9260e31d2d09', 'request_id': 'req-272842f4-7c2b-4c22-aacb-86ce32eae4f8'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:14.962417 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-272842f4-7c2b-4c22-aacb-86ce32eae4f8 admin admin] The token cache is being invalidate because project d0efdd0349e941ef913b9260e31d2d09 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:39:14.962417 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-272842f4-7c2b-4c22-aacb-86ce32eae4f8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:14.968837 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-272842f4-7c2b-4c22-aacb-86ce32eae4f8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:14.971090 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 865/1709] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:39:14 2022] DELETE /identity/v3/projects/d0efdd0349e941ef913b9260e31d2d09 => generated 0 bytes in 730 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:14.975396 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-272842f4-7c2b-4c22-aacb-86ce32eae4f8 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:14.977256 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-272842f4-7c2b-4c22-aacb-86ce32eae4f8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:15.092007 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a378751-44b8-407e-a877-cf8e7201a07d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:15.097832 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a378751-44b8-407e-a877-cf8e7201a07d service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:15.119025 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 845/1710] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:14 2022] GET /identity/v3/auth/tokens => generated 3230 bytes in 393 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:39:15.199442 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-272842f4-7c2b-4c22-aacb-86ce32eae4f8 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:15.206036 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239cba77-cd4c-4628-a95c-5e9c38a60716 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:15.207149 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239cba77-cd4c-4628-a95c-5e9c38a60716 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:15.207502 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239cba77-cd4c-4628-a95c-5e9c38a60716 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:15.207999 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239cba77-cd4c-4628-a95c-5e9c38a60716 admin admin] PATH_INFO: `/v3/users/34de8ce915784775a280db6df99e1965` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:15.209607 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-239cba77-cd4c-4628-a95c-5e9c38a60716 admin admin] RBAC: Authorizing `identity:delete_user(user_id=34de8ce915784775a280db6df99e1965)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:15.211926 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-239cba77-cd4c-4628-a95c-5e9c38a60716 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:15.448594 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a378751-44b8-407e-a877-cf8e7201a07d service cinder] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:15.529059 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-239cba77-cd4c-4628-a95c-5e9c38a60716 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '34de8ce915784775a280db6df99e1965'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:15.529620 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-239cba77-cd4c-4628-a95c-5e9c38a60716 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '34de8ce915784775a280db6df99e1965'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:15.529970 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-239cba77-cd4c-4628-a95c-5e9c38a60716 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '34de8ce915784775a280db6df99e1965'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:15.554347 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-239cba77-cd4c-4628-a95c-5e9c38a60716 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '34de8ce915784775a280db6df99e1965'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:15.590215 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-239cba77-cd4c-4628-a95c-5e9c38a60716 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '34de8ce915784775a280db6df99e1965'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:15.700858 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 866/1711] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:39:14 2022] DELETE /identity/v3/users/34de8ce915784775a280db6df99e1965 => generated 0 bytes in 726 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:39:15.713974 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a378751-44b8-407e-a877-cf8e7201a07d service cinder] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:15.848850 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825ab07a-2923-40e7-a5a5-23ee272b582c tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eb3ab323723400891c7088333bb56e9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0b2661b3862b49629d3ef4002215cd49', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:15.848850 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825ab07a-2923-40e7-a5a5-23ee272b582c tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:15.848850 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825ab07a-2923-40e7-a5a5-23ee272b582c tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:15.849245 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825ab07a-2923-40e7-a5a5-23ee272b582c tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:15.854923 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-825ab07a-2923-40e7-a5a5-23ee272b582c tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:39:15.860267 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-825ab07a-2923-40e7-a5a5-23ee272b582c tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:15.860267 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-825ab07a-2923-40e7-a5a5-23ee272b582c tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:15.924597 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 846/1712] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:39:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 477 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:39:15.928415 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239cba77-cd4c-4628-a95c-5e9c38a60716 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:15.930339 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-239cba77-cd4c-4628-a95c-5e9c38a60716 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:15.959818 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239cba77-cd4c-4628-a95c-5e9c38a60716 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:15.983355 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc27045-7ea7-431b-89d2-93349bfaa3e6 tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eb3ab323723400891c7088333bb56e9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0b2661b3862b49629d3ef4002215cd49', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:15.984897 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc27045-7ea7-431b-89d2-93349bfaa3e6 tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:15.985318 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc27045-7ea7-431b-89d2-93349bfaa3e6 tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:15.985832 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc27045-7ea7-431b-89d2-93349bfaa3e6 tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:15.986669 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc27045-7ea7-431b-89d2-93349bfaa3e6 tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:15.988672 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc27045-7ea7-431b-89d2-93349bfaa3e6 tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:16.171121 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 867/1713] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:39:15 2022] POST /identity/v3/projects => generated 355 bytes in 243 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:16.195962 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825ab07a-2923-40e7-a5a5-23ee272b582c tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:16.200481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:16.254096 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825ab07a-2923-40e7-a5a5-23ee272b582c tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:16.416520 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b974044f-5458-464f-a444-860ed3c8cc69 tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eb3ab323723400891c7088333bb56e9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0b2661b3862b49629d3ef4002215cd49', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:16.419691 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b974044f-5458-464f-a444-860ed3c8cc69 tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:16.420857 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b974044f-5458-464f-a444-860ed3c8cc69 tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:16.421164 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b974044f-5458-464f-a444-860ed3c8cc69 tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:16.425930 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b974044f-5458-464f-a444-860ed3c8cc69 tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:39:16.426638 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b974044f-5458-464f-a444-860ed3c8cc69 tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:16.447034 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1238eb23-8c9d-41fa-a5e0-f709192714d0 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:16.447697 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1238eb23-8c9d-41fa-a5e0-f709192714d0 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:16.448299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1238eb23-8c9d-41fa-a5e0-f709192714d0 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:16.452151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b974044f-5458-464f-a444-860ed3c8cc69 tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:16.460978 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 868/1714] 10.222.0.16 () {54 vars in 912 bytes} [Tue Jul 12 17:39:16 2022] GET /identity => generated 270 bytes in 15 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:39:16.485487 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84643fe-c473-4f75-9994-3e427a812a23 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:16.486250 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84643fe-c473-4f75-9994-3e427a812a23 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:16.486750 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84643fe-c473-4f75-9994-3e427a812a23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:16.490709 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 847/1715] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:39:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 309 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:39:16.612009 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b974044f-5458-464f-a444-860ed3c8cc69 tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:16.612302 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-e84643fe-c473-4f75-9994-3e427a812a23 None None] MFA Rules not processed for user `bf5bf02ae13d4d86a1b157369ef07a00`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:16.614085 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-e84643fe-c473-4f75-9994-3e427a812a23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:16.623398 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b974044f-5458-464f-a444-860ed3c8cc69 tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:16.648230 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d0eb16-ed64-419a-b3e5-e1a0c738a7e1 tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4eb3ab323723400891c7088333bb56e9', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0b2661b3862b49629d3ef4002215cd49', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:16.649212 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d0eb16-ed64-419a-b3e5-e1a0c738a7e1 tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:16.649580 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d0eb16-ed64-419a-b3e5-e1a0c738a7e1 tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:16.649847 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d0eb16-ed64-419a-b3e5-e1a0c738a7e1 tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] PATH_INFO: `/v3/projects/241269621f0945b686660b84d50809dc` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:16.653125 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:16.654473 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d0eb16-ed64-419a-b3e5-e1a0c738a7e1 tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] RBAC: Authorizing `identity:delete_project(project_id=241269621f0945b686660b84d50809dc)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:16.674019 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d0eb16-ed64-419a-b3e5-e1a0c738a7e1 tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:16.790224 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 869/1716] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Jul 12 17:39:16 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 301 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:16.902333 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84643fe-c473-4f75-9994-3e427a812a23 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:16.908239 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-e84643fe-c473-4f75-9994-3e427a812a23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:16.926410 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-90d0eb16-ed64-419a-b3e5-e1a0c738a7e1 tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '241269621f0945b686660b84d50809dc', 'request_id': 'req-90d0eb16-ed64-419a-b3e5-e1a0c738a7e1'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:16.932413 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84643fe-c473-4f75-9994-3e427a812a23 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:16.949604 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-90d0eb16-ed64-419a-b3e5-e1a0c738a7e1 tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '241269621f0945b686660b84d50809dc', 'request_id': 'req-90d0eb16-ed64-419a-b3e5-e1a0c738a7e1'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:16.971301 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-90d0eb16-ed64-419a-b3e5-e1a0c738a7e1 tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '241269621f0945b686660b84d50809dc', 'request_id': 'req-90d0eb16-ed64-419a-b3e5-e1a0c738a7e1'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:17.060380 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8682a3-42a8-486d-9b78-1d28e238e1b4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:17.061349 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8682a3-42a8-486d-9b78-1d28e238e1b4 service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:17.063375 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8682a3-42a8-486d-9b78-1d28e238e1b4 service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:17.063989 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8682a3-42a8-486d-9b78-1d28e238e1b4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:17.074136 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-90d0eb16-ed64-419a-b3e5-e1a0c738a7e1 tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] The token cache is being invalidate because project 241269621f0945b686660b84d50809dc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:39:17.076967 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-90d0eb16-ed64-419a-b3e5-e1a0c738a7e1 tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:17.077857 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-90d0eb16-ed64-419a-b3e5-e1a0c738a7e1 tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:17.080479 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 848/1717] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:39:16 2022] DELETE /identity/v3/projects/241269621f0945b686660b84d50809dc => generated 0 bytes in 469 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:17.120799 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8682a3-42a8-486d-9b78-1d28e238e1b4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:17.127069 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8682a3-42a8-486d-9b78-1d28e238e1b4 service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:17.171181 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 870/1718] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:16 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 269 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:39:17.211665 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:17.625198 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:17.667458 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d0eb16-ed64-419a-b3e5-e1a0c738a7e1 tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:17.671795 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-90d0eb16-ed64-419a-b3e5-e1a0c738a7e1 tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:17.736749 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:17.860614 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5d5327-95da-4342-992a-28a757104ca9 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:17.860905 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5d5327-95da-4342-992a-28a757104ca9 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:17.861188 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5d5327-95da-4342-992a-28a757104ca9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:17.909824 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d0eb16-ed64-419a-b3e5-e1a0c738a7e1 tempest-VolumeQuotaClassesTest-25331259 tempest-VolumeQuotaClassesTest-25331259-project-admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:17.924990 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a7bb3f-8bff-4d71-82d2-bf92e0f17421 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:17.926857 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a7bb3f-8bff-4d71-82d2-bf92e0f17421 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:17.957629 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a7bb3f-8bff-4d71-82d2-bf92e0f17421 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:17.958134 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a7bb3f-8bff-4d71-82d2-bf92e0f17421 admin admin] PATH_INFO: `/v3/projects/95e70964673a48d092a7568dc6927b89` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:17.961114 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a7bb3f-8bff-4d71-82d2-bf92e0f17421 admin admin] RBAC: Authorizing `identity:delete_project(project_id=95e70964673a48d092a7568dc6927b89)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:17.963445 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a7bb3f-8bff-4d71-82d2-bf92e0f17421 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:17.988496 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-ab5d5327-95da-4342-992a-28a757104ca9 None None] MFA Rules not processed for user `9feeaf257bb441b7a14b57f1f493d08f`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:17.991147 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-ab5d5327-95da-4342-992a-28a757104ca9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:18.096385 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 871/1719] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:17 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 237 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:18.097555 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5d5327-95da-4342-992a-28a757104ca9 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:18.100060 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-ab5d5327-95da-4342-992a-28a757104ca9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:18.127048 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5d5327-95da-4342-992a-28a757104ca9 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:18.132232 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ad165e-a9ed-4ebd-ae6b-59045f0e3445 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:18.133189 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ad165e-a9ed-4ebd-ae6b-59045f0e3445 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:18.133658 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ad165e-a9ed-4ebd-ae6b-59045f0e3445 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:18.134185 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ad165e-a9ed-4ebd-ae6b-59045f0e3445 admin admin] PATH_INFO: `/v3/users/f18be44ed7f64d4fae92dc6d79a84dc6` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:18.135756 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5ad165e-a9ed-4ebd-ae6b-59045f0e3445 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f18be44ed7f64d4fae92dc6d79a84dc6)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:18.142284 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5ad165e-a9ed-4ebd-ae6b-59045f0e3445 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:18.252804 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-33a7bb3f-8bff-4d71-82d2-bf92e0f17421 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '95e70964673a48d092a7568dc6927b89', 'request_id': 'req-33a7bb3f-8bff-4d71-82d2-bf92e0f17421'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:18.275002 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-33a7bb3f-8bff-4d71-82d2-bf92e0f17421 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '95e70964673a48d092a7568dc6927b89', 'request_id': 'req-33a7bb3f-8bff-4d71-82d2-bf92e0f17421'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:18.283368 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-33a7bb3f-8bff-4d71-82d2-bf92e0f17421 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '95e70964673a48d092a7568dc6927b89', 'request_id': 'req-33a7bb3f-8bff-4d71-82d2-bf92e0f17421'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:18.374599 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-a5ad165e-a9ed-4ebd-ae6b-59045f0e3445 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f18be44ed7f64d4fae92dc6d79a84dc6'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:18.376638 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-a5ad165e-a9ed-4ebd-ae6b-59045f0e3445 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f18be44ed7f64d4fae92dc6d79a84dc6'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:18.385627 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-33a7bb3f-8bff-4d71-82d2-bf92e0f17421 admin admin] The token cache is being invalidate because project 95e70964673a48d092a7568dc6927b89 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:39:18.386134 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-33a7bb3f-8bff-4d71-82d2-bf92e0f17421 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:18.387235 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-33a7bb3f-8bff-4d71-82d2-bf92e0f17421 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:18.389429 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-a5ad165e-a9ed-4ebd-ae6b-59045f0e3445 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f18be44ed7f64d4fae92dc6d79a84dc6'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:18.392707 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 849/1720] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:39:17 2022] DELETE /identity/v3/projects/95e70964673a48d092a7568dc6927b89 => generated 0 bytes in 726 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:18.393952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a7bb3f-8bff-4d71-82d2-bf92e0f17421 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:18.397967 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-33a7bb3f-8bff-4d71-82d2-bf92e0f17421 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:18.415989 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-a5ad165e-a9ed-4ebd-ae6b-59045f0e3445 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f18be44ed7f64d4fae92dc6d79a84dc6'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:18.475488 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:18.477263 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-a5ad165e-a9ed-4ebd-ae6b-59045f0e3445 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f18be44ed7f64d4fae92dc6d79a84dc6'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:18.554133 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 872/1721] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:39:18 2022] DELETE /identity/v3/users/f18be44ed7f64d4fae92dc6d79a84dc6 => generated 0 bytes in 454 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:18.554133 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ad165e-a9ed-4ebd-ae6b-59045f0e3445 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:18.558143 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-a5ad165e-a9ed-4ebd-ae6b-59045f0e3445 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:18.623810 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a7bb3f-8bff-4d71-82d2-bf92e0f17421 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:18.725993 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ad165e-a9ed-4ebd-ae6b-59045f0e3445 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:18.735297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f47099-84f9-48c8-a609-d2dfbac41d8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:18.736611 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f47099-84f9-48c8-a609-d2dfbac41d8e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:18.737442 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f47099-84f9-48c8-a609-d2dfbac41d8e admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:18.738234 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f47099-84f9-48c8-a609-d2dfbac41d8e admin admin] PATH_INFO: `/v3/users/ceac25e8a7c849c7a578c3ac762fe824` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:18.741887 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f47099-84f9-48c8-a609-d2dfbac41d8e admin admin] RBAC: Authorizing `identity:delete_user(user_id=ceac25e8a7c849c7a578c3ac762fe824)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:18.746574 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f47099-84f9-48c8-a609-d2dfbac41d8e admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:18.770872 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3125b75e-3599-425e-a491-6bc973564e45 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:18.772227 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3125b75e-3599-425e-a491-6bc973564e45 service nova] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:18.772686 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3125b75e-3599-425e-a491-6bc973564e45 service nova] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:18.773116 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3125b75e-3599-425e-a491-6bc973564e45 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:18.780023 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-3125b75e-3599-425e-a491-6bc973564e45 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:18.897678 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3125b75e-3599-425e-a491-6bc973564e45 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:18.900822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3125b75e-3599-425e-a491-6bc973564e45 service nova] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:18.925153 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 850/1722] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:18 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 532 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:39:18.926915 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683340a9-cf2a-436d-9cb7-daa0ec3db184 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:18.928299 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683340a9-cf2a-436d-9cb7-daa0ec3db184 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:18.928879 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683340a9-cf2a-436d-9cb7-daa0ec3db184 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:18.960798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:18.987052 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-683340a9-cf2a-436d-9cb7-daa0ec3db184 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:18.988337 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-683340a9-cf2a-436d-9cb7-daa0ec3db184 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:19.018129 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 851/1723] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:18 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:19.037401 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683340a9-cf2a-436d-9cb7-daa0ec3db184 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:19.039187 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-683340a9-cf2a-436d-9cb7-daa0ec3db184 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:19.070313 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:19.100019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-79f47099-84f9-48c8-a609-d2dfbac41d8e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ceac25e8a7c849c7a578c3ac762fe824'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:19.100807 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-79f47099-84f9-48c8-a609-d2dfbac41d8e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ceac25e8a7c849c7a578c3ac762fe824'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:19.102856 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-79f47099-84f9-48c8-a609-d2dfbac41d8e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ceac25e8a7c849c7a578c3ac762fe824'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:19.135134 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-79f47099-84f9-48c8-a609-d2dfbac41d8e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ceac25e8a7c849c7a578c3ac762fe824'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:19.173753 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-79f47099-84f9-48c8-a609-d2dfbac41d8e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ceac25e8a7c849c7a578c3ac762fe824'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:19.230825 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683340a9-cf2a-436d-9cb7-daa0ec3db184 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:19.235838 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51854d67-54d8-4f52-89d9-c0cf8a9558a3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:19.236534 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51854d67-54d8-4f52-89d9-c0cf8a9558a3 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:19.236811 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51854d67-54d8-4f52-89d9-c0cf8a9558a3 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:19.237096 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51854d67-54d8-4f52-89d9-c0cf8a9558a3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:19.239151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-51854d67-54d8-4f52-89d9-c0cf8a9558a3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:19.255872 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51854d67-54d8-4f52-89d9-c0cf8a9558a3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:19.257506 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51854d67-54d8-4f52-89d9-c0cf8a9558a3 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:19.270185 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 852/1724] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:19 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 233 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:39:19.271783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304a6c4f-6de3-4daa-8540-817fb77dbf2c None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:19.272086 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304a6c4f-6de3-4daa-8540-817fb77dbf2c None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:19.272408 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304a6c4f-6de3-4daa-8540-817fb77dbf2c None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:19.274113 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 853/1725] 10.222.0.16 () {54 vars in 921 bytes} [Tue Jul 12 17:39:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:39:19.300088 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 873/1726] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:39:18 2022] DELETE /identity/v3/users/ceac25e8a7c849c7a578c3ac762fe824 => generated 0 bytes in 748 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:19.325894 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1dacaa-28a1-4891-8e29-636ad8b48bc2 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:19.331276 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1dacaa-28a1-4891-8e29-636ad8b48bc2 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:19.331276 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1dacaa-28a1-4891-8e29-636ad8b48bc2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:19.441843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-4b1dacaa-28a1-4891-8e29-636ad8b48bc2 None None] MFA Rules not processed for user `f623be28c83247c78d3ea6873c347fe0`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:19.441843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-4b1dacaa-28a1-4891-8e29-636ad8b48bc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:19.589862 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c7da8d-d2aa-404d-bf89-5c8a5fc3d1c3 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:19.590618 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c7da8d-d2aa-404d-bf89-5c8a5fc3d1c3 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:19.591192 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c7da8d-d2aa-404d-bf89-5c8a5fc3d1c3 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:19.593502 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 874/1727] 10.222.0.16 () {54 vars in 960 bytes} [Tue Jul 12 17:39:19 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 12 17:39:19.599771 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3251b4a5-bb9f-42dc-b166-eb146f2e08bb None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:19.601413 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3251b4a5-bb9f-42dc-b166-eb146f2e08bb None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:19.601799 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3251b4a5-bb9f-42dc-b166-eb146f2e08bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:19.633794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 854/1728] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Jul 12 17:39:19 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 309 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:19.639061 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-3251b4a5-bb9f-42dc-b166-eb146f2e08bb None None] MFA Rules not processed for user `f623be28c83247c78d3ea6873c347fe0`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:19.642098 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-3251b4a5-bb9f-42dc-b166-eb146f2e08bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:19.677178 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 875/1729] 10.222.0.16 () {58 vars in 1057 bytes} [Tue Jul 12 17:39:19 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:39:19.686092 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1dacaa-28a1-4891-8e29-636ad8b48bc2 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:19.701857 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1dacaa-28a1-4891-8e29-636ad8b48bc2 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:19.722855 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9d6204-e3d7-4906-9296-62a7696de1ba service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f623be28c83247c78d3ea6873c347fe0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:19.723824 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9d6204-e3d7-4906-9296-62a7696de1ba service placement] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:19.724212 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9d6204-e3d7-4906-9296-62a7696de1ba service placement] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:19.724597 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9d6204-e3d7-4906-9296-62a7696de1ba service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:19.740087 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-6c9d6204-e3d7-4906-9296-62a7696de1ba service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:19.778179 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:19.780437 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3251b4a5-bb9f-42dc-b166-eb146f2e08bb None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:19.783151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c9d6204-e3d7-4906-9296-62a7696de1ba service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:19.785100 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c9d6204-e3d7-4906-9296-62a7696de1ba service placement] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:19.787723 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-3251b4a5-bb9f-42dc-b166-eb146f2e08bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:19.807348 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3251b4a5-bb9f-42dc-b166-eb146f2e08bb None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:19.822289 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 855/1730] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:19 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:39:19.832736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4518623-b5a8-40c6-8f28-edb4779c21b3 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f623be28c83247c78d3ea6873c347fe0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:19.833885 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4518623-b5a8-40c6-8f28-edb4779c21b3 service placement] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:19.834377 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4518623-b5a8-40c6-8f28-edb4779c21b3 service placement] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:19.836486 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4518623-b5a8-40c6-8f28-edb4779c21b3 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:19.848117 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4518623-b5a8-40c6-8f28-edb4779c21b3 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:19.850161 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4518623-b5a8-40c6-8f28-edb4779c21b3 service placement] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:19.863970 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 876/1731] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:19 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:39:19.929377 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23b01cd-7590-4112-bee1-cd03f1996495 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:19.929377 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23b01cd-7590-4112-bee1-cd03f1996495 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:19.929377 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23b01cd-7590-4112-bee1-cd03f1996495 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:20.003477 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-f23b01cd-7590-4112-bee1-cd03f1996495 None None] MFA Rules not processed for user `c8441a4a72154ee3bbb95af66abf2ffd`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:20.004760 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-f23b01cd-7590-4112-bee1-cd03f1996495 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:20.194640 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 856/1732] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:19 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 269 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:20.210387 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4518623-b5a8-40c6-8f28-edb4779c21b3 service placement] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:20.212294 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-f4518623-b5a8-40c6-8f28-edb4779c21b3 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:20.269375 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23b01cd-7590-4112-bee1-cd03f1996495 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:20.270962 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-f23b01cd-7590-4112-bee1-cd03f1996495 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:20.370061 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4518623-b5a8-40c6-8f28-edb4779c21b3 service placement] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:20.400235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03acc833-4cfb-4c40-9eb4-b1948844a5b5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:20.401137 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03acc833-4cfb-4c40-9eb4-b1948844a5b5 service nova] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:20.402844 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03acc833-4cfb-4c40-9eb4-b1948844a5b5 service nova] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:20.418463 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03acc833-4cfb-4c40-9eb4-b1948844a5b5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:20.425584 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03acc833-4cfb-4c40-9eb4-b1948844a5b5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:20.427782 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03acc833-4cfb-4c40-9eb4-b1948844a5b5 service nova] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:20.432433 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23b01cd-7590-4112-bee1-cd03f1996495 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:20.441291 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1884315d-741d-4386-a960-a48c4a9e52b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:20.442563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1884315d-741d-4386-a960-a48c4a9e52b2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:20.443027 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1884315d-741d-4386-a960-a48c4a9e52b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:20.443956 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 877/1733] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:20 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 234 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:39:20.446111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2906094-409f-4a60-9e22-a6a43b7905a1 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:20.446612 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2906094-409f-4a60-9e22-a6a43b7905a1 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:20.447140 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2906094-409f-4a60-9e22-a6a43b7905a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:20.454524 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1884315d-741d-4386-a960-a48c4a9e52b2 admin admin] PATH_INFO: `/v3/users/aad7479f671a4d0da933c72688d867ba` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:20.460313 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1884315d-741d-4386-a960-a48c4a9e52b2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=aad7479f671a4d0da933c72688d867ba)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:20.467611 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1884315d-741d-4386-a960-a48c4a9e52b2 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:20.494768 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-d2906094-409f-4a60-9e22-a6a43b7905a1 None None] MFA Rules not processed for user `ce09df98c929416e8ca67b0728349bfb`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:20.497312 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-d2906094-409f-4a60-9e22-a6a43b7905a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:20.565472 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:20.620305 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 878/1734] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:20 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:20.757682 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-1884315d-741d-4386-a960-a48c4a9e52b2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'aad7479f671a4d0da933c72688d867ba'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:20.893946 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-1884315d-741d-4386-a960-a48c4a9e52b2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'aad7479f671a4d0da933c72688d867ba'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:20.895081 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-1884315d-741d-4386-a960-a48c4a9e52b2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'aad7479f671a4d0da933c72688d867ba'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:20.912992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-1884315d-741d-4386-a960-a48c4a9e52b2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'aad7479f671a4d0da933c72688d867ba'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:21.001202 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-1884315d-741d-4386-a960-a48c4a9e52b2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'aad7479f671a4d0da933c72688d867ba'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:21.013384 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 857/1735] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:39:20 2022] DELETE /identity/v3/users/aad7479f671a4d0da933c72688d867ba => generated 0 bytes in 745 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:39:21.019900 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5113b04a-d672-444d-8d44-52dc2d6bf4be None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:21.020853 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5113b04a-d672-444d-8d44-52dc2d6bf4be None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:21.021353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5113b04a-d672-444d-8d44-52dc2d6bf4be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:21.052237 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-5113b04a-d672-444d-8d44-52dc2d6bf4be None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:21.054788 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-5113b04a-d672-444d-8d44-52dc2d6bf4be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:21.194396 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 879/1736] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:21.217539 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1884315d-741d-4386-a960-a48c4a9e52b2 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:21.220383 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-1884315d-741d-4386-a960-a48c4a9e52b2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:21.310937 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=98,keystone:INSERT=1,keystone:DELETE=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:21.395542 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1884315d-741d-4386-a960-a48c4a9e52b2 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:21.411656 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661902bb-58f9-46dc-9542-06597d1e8fce service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:21.415281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661902bb-58f9-46dc-9542-06597d1e8fce service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:21.415969 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661902bb-58f9-46dc-9542-06597d1e8fce service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:21.417446 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661902bb-58f9-46dc-9542-06597d1e8fce service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:21.437372 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-661902bb-58f9-46dc-9542-06597d1e8fce service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:21.441533 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-661902bb-58f9-46dc-9542-06597d1e8fce service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:21.465938 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 858/1737] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:21 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 249 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:39:21.600496 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99633b9b-c809-4f09-a94b-4a683bbfbcb6 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:21.601247 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99633b9b-c809-4f09-a94b-4a683bbfbcb6 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:21.601939 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99633b9b-c809-4f09-a94b-4a683bbfbcb6 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:21.611934 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 880/1738] 10.222.0.16 () {54 vars in 912 bytes} [Tue Jul 12 17:39:21 2022] GET /identity => generated 270 bytes in 14 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:39:21.625450 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d65c5a-3a2c-44c7-afdf-2437ada3b7f4 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:21.625799 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d65c5a-3a2c-44c7-afdf-2437ada3b7f4 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:21.626084 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d65c5a-3a2c-44c7-afdf-2437ada3b7f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:21.644006 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99633b9b-c809-4f09-a94b-4a683bbfbcb6 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:21.647120 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-99633b9b-c809-4f09-a94b-4a683bbfbcb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:21.673778 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-73d65c5a-3a2c-44c7-afdf-2437ada3b7f4 None None] MFA Rules not processed for user `bf5bf02ae13d4d86a1b157369ef07a00`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:21.674956 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99633b9b-c809-4f09-a94b-4a683bbfbcb6 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:21.677404 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-73d65c5a-3a2c-44c7-afdf-2437ada3b7f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:21.683092 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b44f43-9d63-43f5-844f-7e7d1c93a16a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:21.684487 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b44f43-9d63-43f5-844f-7e7d1c93a16a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:21.685165 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b44f43-9d63-43f5-844f-7e7d1c93a16a admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:21.685595 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b44f43-9d63-43f5-844f-7e7d1c93a16a admin admin] PATH_INFO: `/v3/projects/1a54f9fed45146cbaa39ecd1f6df41cc` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:21.689191 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b44f43-9d63-43f5-844f-7e7d1c93a16a admin admin] RBAC: Authorizing `identity:delete_project(project_id=1a54f9fed45146cbaa39ecd1f6df41cc)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:21.696208 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b44f43-9d63-43f5-844f-7e7d1c93a16a admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:21.696606 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 859/1739] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Jul 12 17:39:21 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 70 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:21.713160 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d65c5a-3a2c-44c7-afdf-2437ada3b7f4 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:21.732947 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-73d65c5a-3a2c-44c7-afdf-2437ada3b7f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:21.862335 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d65c5a-3a2c-44c7-afdf-2437ada3b7f4 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:21.920873 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5a79c0-7384-4652-b9bb-81b7e7f96ae7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:21.932001 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5a79c0-7384-4652-b9bb-81b7e7f96ae7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:21.932713 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5a79c0-7384-4652-b9bb-81b7e7f96ae7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:21.944089 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5a79c0-7384-4652-b9bb-81b7e7f96ae7 admin admin] PATH_INFO: `/v3/projects/78a7fac0a1ad4f098766b812b54647a9` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:21.954185 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5a79c0-7384-4652-b9bb-81b7e7f96ae7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=78a7fac0a1ad4f098766b812b54647a9)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:21.969140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5a79c0-7384-4652-b9bb-81b7e7f96ae7 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:22.152647 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e6b44f43-9d63-43f5-844f-7e7d1c93a16a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1a54f9fed45146cbaa39ecd1f6df41cc', 'request_id': 'req-e6b44f43-9d63-43f5-844f-7e7d1c93a16a'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:22.181186 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e6b44f43-9d63-43f5-844f-7e7d1c93a16a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1a54f9fed45146cbaa39ecd1f6df41cc', 'request_id': 'req-e6b44f43-9d63-43f5-844f-7e7d1c93a16a'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:22.255530 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-9c5a79c0-7384-4652-b9bb-81b7e7f96ae7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '78a7fac0a1ad4f098766b812b54647a9', 'request_id': 'req-9c5a79c0-7384-4652-b9bb-81b7e7f96ae7'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:22.266787 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-9c5a79c0-7384-4652-b9bb-81b7e7f96ae7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '78a7fac0a1ad4f098766b812b54647a9', 'request_id': 'req-9c5a79c0-7384-4652-b9bb-81b7e7f96ae7'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:22.282367 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-9c5a79c0-7384-4652-b9bb-81b7e7f96ae7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '78a7fac0a1ad4f098766b812b54647a9', 'request_id': 'req-9c5a79c0-7384-4652-b9bb-81b7e7f96ae7'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:22.286934 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:22.334229 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e6b44f43-9d63-43f5-844f-7e7d1c93a16a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1a54f9fed45146cbaa39ecd1f6df41cc', 'request_id': 'req-e6b44f43-9d63-43f5-844f-7e7d1c93a16a'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:22.417957 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e6b44f43-9d63-43f5-844f-7e7d1c93a16a admin admin] The token cache is being invalidate because project 1a54f9fed45146cbaa39ecd1f6df41cc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:39:22.418504 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e6b44f43-9d63-43f5-844f-7e7d1c93a16a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:22.419774 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e6b44f43-9d63-43f5-844f-7e7d1c93a16a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:22.423653 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 881/1740] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:39:21 2022] DELETE /identity/v3/projects/1a54f9fed45146cbaa39ecd1f6df41cc => generated 0 bytes in 780 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:22.424694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b44f43-9d63-43f5-844f-7e7d1c93a16a admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:22.437199 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-9c5a79c0-7384-4652-b9bb-81b7e7f96ae7 admin admin] The token cache is being invalidate because project 78a7fac0a1ad4f098766b812b54647a9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:39:22.438463 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-9c5a79c0-7384-4652-b9bb-81b7e7f96ae7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:22.438898 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-e6b44f43-9d63-43f5-844f-7e7d1c93a16a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:22.439047 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-9c5a79c0-7384-4652-b9bb-81b7e7f96ae7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:22.441939 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 860/1741] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:39:21 2022] DELETE /identity/v3/projects/78a7fac0a1ad4f098766b812b54647a9 => generated 0 bytes in 732 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:22.442876 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5a79c0-7384-4652-b9bb-81b7e7f96ae7 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:22.453349 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-9c5a79c0-7384-4652-b9bb-81b7e7f96ae7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:22.600995 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5a79c0-7384-4652-b9bb-81b7e7f96ae7 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:22.608923 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98bcc2f8-33cc-4b47-805e-190c0830c88a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:22.611012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98bcc2f8-33cc-4b47-805e-190c0830c88a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:22.611544 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98bcc2f8-33cc-4b47-805e-190c0830c88a admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:22.612137 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98bcc2f8-33cc-4b47-805e-190c0830c88a admin admin] PATH_INFO: `/v3/users/4eb3ab323723400891c7088333bb56e9` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:22.615821 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98bcc2f8-33cc-4b47-805e-190c0830c88a admin admin] RBAC: Authorizing `identity:delete_user(user_id=4eb3ab323723400891c7088333bb56e9)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:22.621551 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98bcc2f8-33cc-4b47-805e-190c0830c88a admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:22.714766 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b44f43-9d63-43f5-844f-7e7d1c93a16a admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:22.963247 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-98bcc2f8-33cc-4b47-805e-190c0830c88a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4eb3ab323723400891c7088333bb56e9'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:22.994576 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0266b78-358f-4119-84b6-9524a8830c00 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:23.000310 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0266b78-358f-4119-84b6-9524a8830c00 service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:23.002094 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0266b78-358f-4119-84b6-9524a8830c00 service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:23.008663 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0266b78-358f-4119-84b6-9524a8830c00 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:23.020351 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-b0266b78-358f-4119-84b6-9524a8830c00 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:23.049618 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0266b78-358f-4119-84b6-9524a8830c00 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:23.054356 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0266b78-358f-4119-84b6-9524a8830c00 service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:23.064288 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-98bcc2f8-33cc-4b47-805e-190c0830c88a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4eb3ab323723400891c7088333bb56e9'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:23.065324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-98bcc2f8-33cc-4b47-805e-190c0830c88a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4eb3ab323723400891c7088333bb56e9'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:23.098355 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-98bcc2f8-33cc-4b47-805e-190c0830c88a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4eb3ab323723400891c7088333bb56e9'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:23.123763 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 882/1742] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:22 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 700 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:39:23.131556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5f39f8-18f6-473d-8bbc-6ca66ca344fd None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:23.132082 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5f39f8-18f6-473d-8bbc-6ca66ca344fd None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:23.132603 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5f39f8-18f6-473d-8bbc-6ca66ca344fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:23.143138 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-98bcc2f8-33cc-4b47-805e-190c0830c88a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4eb3ab323723400891c7088333bb56e9'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:23.157845 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 861/1743] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:39:22 2022] DELETE /identity/v3/users/4eb3ab323723400891c7088333bb56e9 => generated 0 bytes in 715 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:23.164257 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98bcc2f8-33cc-4b47-805e-190c0830c88a admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:23.164257 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-98bcc2f8-33cc-4b47-805e-190c0830c88a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:23.190514 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:23.264468 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-0b5f39f8-18f6-473d-8bbc-6ca66ca344fd None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:23.270094 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-0b5f39f8-18f6-473d-8bbc-6ca66ca344fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:23.305091 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98bcc2f8-33cc-4b47-805e-190c0830c88a admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:23.311817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58d2d70-4c2f-41e0-85a9-0fb9aee2adc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:23.313457 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58d2d70-4c2f-41e0-85a9-0fb9aee2adc7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:23.313855 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58d2d70-4c2f-41e0-85a9-0fb9aee2adc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:23.314337 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58d2d70-4c2f-41e0-85a9-0fb9aee2adc7 admin admin] PATH_INFO: `/v3/users/18a8a578f3554b3dbf53a50db6c94faf` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:23.316289 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58d2d70-4c2f-41e0-85a9-0fb9aee2adc7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=18a8a578f3554b3dbf53a50db6c94faf)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:23.319093 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d58d2d70-4c2f-41e0-85a9-0fb9aee2adc7 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:23.428339 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 883/1744] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 304 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:23.440183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250e10db-8925-41c1-921b-52d3b8b3c7d1 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:23.440655 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250e10db-8925-41c1-921b-52d3b8b3c7d1 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:23.441022 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250e10db-8925-41c1-921b-52d3b8b3c7d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:23.504005 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-250e10db-8925-41c1-921b-52d3b8b3c7d1 None None] MFA Rules not processed for user `cd79a28d8f774c1fac44185df36cd24f`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:23.507056 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-250e10db-8925-41c1-921b-52d3b8b3c7d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:23.651957 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 884/1745] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:23 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:23.657245 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5032941c-4592-47e7-9b17-2dbae868d43c None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:23.657628 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5032941c-4592-47e7-9b17-2dbae868d43c None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:23.657925 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5032941c-4592-47e7-9b17-2dbae868d43c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:23.681616 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-d58d2d70-4c2f-41e0-85a9-0fb9aee2adc7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '18a8a578f3554b3dbf53a50db6c94faf'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:23.708052 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-5032941c-4592-47e7-9b17-2dbae868d43c None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:23.709382 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-5032941c-4592-47e7-9b17-2dbae868d43c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:23.729588 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 885/1746] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:23.731247 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fcb3c5-e679-47d0-8a11-5809daa9b172 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:23.731674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fcb3c5-e679-47d0-8a11-5809daa9b172 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:23.732107 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fcb3c5-e679-47d0-8a11-5809daa9b172 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:23.777664 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-c4fcb3c5-e679-47d0-8a11-5809daa9b172 None None] MFA Rules not processed for user `cd79a28d8f774c1fac44185df36cd24f`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:23.780142 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-c4fcb3c5-e679-47d0-8a11-5809daa9b172 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:23.809215 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 886/1747] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:23 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:23.809215 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fcb3c5-e679-47d0-8a11-5809daa9b172 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:23.830325 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-d58d2d70-4c2f-41e0-85a9-0fb9aee2adc7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '18a8a578f3554b3dbf53a50db6c94faf'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:23.831289 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-d58d2d70-4c2f-41e0-85a9-0fb9aee2adc7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '18a8a578f3554b3dbf53a50db6c94faf'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:23.831798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fcb3c5-e679-47d0-8a11-5809daa9b172 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:23.842890 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1937fa-0011-44d2-9df2-7c5e10a141a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:23.844086 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1937fa-0011-44d2-9df2-7c5e10a141a3 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:23.844528 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1937fa-0011-44d2-9df2-7c5e10a141a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:23.844950 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1937fa-0011-44d2-9df2-7c5e10a141a3 admin admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:23.845993 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a1937fa-0011-44d2-9df2-7c5e10a141a3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:39:23.846494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a1937fa-0011-44d2-9df2-7c5e10a141a3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:23.848288 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a1937fa-0011-44d2-9df2-7c5e10a141a3 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:23.849060 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-d58d2d70-4c2f-41e0-85a9-0fb9aee2adc7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '18a8a578f3554b3dbf53a50db6c94faf'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:23.878501 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 887/1748] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:39:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:39:23.880978 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1937fa-0011-44d2-9df2-7c5e10a141a3 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:23.887733 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-7a1937fa-0011-44d2-9df2-7c5e10a141a3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:23.942589 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-d58d2d70-4c2f-41e0-85a9-0fb9aee2adc7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '18a8a578f3554b3dbf53a50db6c94faf'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:23.975346 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 862/1749] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:39:23 2022] DELETE /identity/v3/users/18a8a578f3554b3dbf53a50db6c94faf => generated 0 bytes in 814 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:23.977994 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58d2d70-4c2f-41e0-85a9-0fb9aee2adc7 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:23.989140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-d58d2d70-4c2f-41e0-85a9-0fb9aee2adc7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:24.041243 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:24.133435 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:24.162359 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1937fa-0011-44d2-9df2-7c5e10a141a3 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:24.310798 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d58d2d70-4c2f-41e0-85a9-0fb9aee2adc7 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:24.325069 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d8b004-51cc-48ea-a732-34902c963d01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:24.325948 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d8b004-51cc-48ea-a732-34902c963d01 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:24.326831 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d8b004-51cc-48ea-a732-34902c963d01 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:24.327747 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5d8b004-51cc-48ea-a732-34902c963d01 admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:24.331695 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d8b004-51cc-48ea-a732-34902c963d01 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:24.337249 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5d8b004-51cc-48ea-a732-34902c963d01 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:24.366643 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b027e8-8014-47fa-b929-170384dd430f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:24.367336 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b027e8-8014-47fa-b929-170384dd430f service nova] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:24.367652 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b027e8-8014-47fa-b929-170384dd430f service nova] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:24.374275 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b027e8-8014-47fa-b929-170384dd430f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:24.384068 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-71b027e8-8014-47fa-b929-170384dd430f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:24.546285 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 863/1750] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:39:23 2022] POST /identity/v3/projects => generated 373 bytes in 570 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:24.547789 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d8b004-51cc-48ea-a732-34902c963d01 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:24.552179 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-a5d8b004-51cc-48ea-a732-34902c963d01 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:24.709842 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b027e8-8014-47fa-b929-170384dd430f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:24.720766 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b027e8-8014-47fa-b929-170384dd430f service nova] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:24.746774 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 888/1751] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:23 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 866 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:39:24.756421 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b027e8-8014-47fa-b929-170384dd430f service nova] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:24.817754 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b027e8-8014-47fa-b929-170384dd430f service nova] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:24.870599 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5d8b004-51cc-48ea-a732-34902c963d01 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:24.893680 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f0e393-1537-4135-ba18-6f50b31647e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:24.899560 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f0e393-1537-4135-ba18-6f50b31647e4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:24.900147 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f0e393-1537-4135-ba18-6f50b31647e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:24.900572 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f0e393-1537-4135-ba18-6f50b31647e4 admin admin] PATH_INFO: `/v3/projects/0d9858aaa86e461a9e13a9521a562c2b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:24.906385 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f0e393-1537-4135-ba18-6f50b31647e4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0d9858aaa86e461a9e13a9521a562c2b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:24.906385 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f0e393-1537-4135-ba18-6f50b31647e4 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:25.122748 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-b2f0e393-1537-4135-ba18-6f50b31647e4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0d9858aaa86e461a9e13a9521a562c2b', 'request_id': 'req-b2f0e393-1537-4135-ba18-6f50b31647e4'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:25.169875 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-b2f0e393-1537-4135-ba18-6f50b31647e4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0d9858aaa86e461a9e13a9521a562c2b', 'request_id': 'req-b2f0e393-1537-4135-ba18-6f50b31647e4'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:25.195702 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f9dc8a1-4808-406d-8f65-826b31b4bfda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:25.197756 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9dc8a1-4808-406d-8f65-826b31b4bfda admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:25.198322 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9dc8a1-4808-406d-8f65-826b31b4bfda admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:25.199830 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9dc8a1-4808-406d-8f65-826b31b4bfda admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:25.201925 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f9dc8a1-4808-406d-8f65-826b31b4bfda admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:25.209319 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f9dc8a1-4808-406d-8f65-826b31b4bfda admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:25.216641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-b2f0e393-1537-4135-ba18-6f50b31647e4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0d9858aaa86e461a9e13a9521a562c2b', 'request_id': 'req-b2f0e393-1537-4135-ba18-6f50b31647e4'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:25.382279 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-b2f0e393-1537-4135-ba18-6f50b31647e4 admin admin] The token cache is being invalidate because project 0d9858aaa86e461a9e13a9521a562c2b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:39:25.382790 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-b2f0e393-1537-4135-ba18-6f50b31647e4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:25.384532 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-b2f0e393-1537-4135-ba18-6f50b31647e4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:25.387629 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 864/1752] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:39:24 2022] DELETE /identity/v3/projects/0d9858aaa86e461a9e13a9521a562c2b => generated 0 bytes in 841 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:25.393522 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f0e393-1537-4135-ba18-6f50b31647e4 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:25.401601 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-b2f0e393-1537-4135-ba18-6f50b31647e4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:25.434982 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 889/1753] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:39:24 2022] POST /identity/v3/users => generated 287 bytes in 687 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:25.442092 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f9dc8a1-4808-406d-8f65-826b31b4bfda admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:25.443625 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-0f9dc8a1-4808-406d-8f65-826b31b4bfda admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:25.479732 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f9dc8a1-4808-406d-8f65-826b31b4bfda admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:25.626986 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f3ae01-0976-4200-a9f0-172c638bb21e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:25.628362 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f3ae01-0976-4200-a9f0-172c638bb21e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:25.628952 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f3ae01-0976-4200-a9f0-172c638bb21e admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:25.629444 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f3ae01-0976-4200-a9f0-172c638bb21e admin admin] PATH_INFO: `/v3/users/a48a3de2cdec4911bc802036e5393302` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:25.631364 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f3ae01-0976-4200-a9f0-172c638bb21e admin admin] RBAC: Authorizing `identity:delete_user(user_id=a48a3de2cdec4911bc802036e5393302)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:25.634674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f3ae01-0976-4200-a9f0-172c638bb21e admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:25.712827 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f0e393-1537-4135-ba18-6f50b31647e4 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:25.731892 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e519326-a991-40a9-8e43-8a78f14ee912 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:25.733563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e519326-a991-40a9-8e43-8a78f14ee912 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:25.738955 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e519326-a991-40a9-8e43-8a78f14ee912 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:25.739771 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e519326-a991-40a9-8e43-8a78f14ee912 admin admin] PATH_INFO: `/v3/projects/0fa254f4eb59487a91bd5807aca4028a` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:25.742403 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e519326-a991-40a9-8e43-8a78f14ee912 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0fa254f4eb59487a91bd5807aca4028a)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:25.750695 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e519326-a991-40a9-8e43-8a78f14ee912 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:26.029328 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-c0f3ae01-0976-4200-a9f0-172c638bb21e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a48a3de2cdec4911bc802036e5393302'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:26.031436 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-c0f3ae01-0976-4200-a9f0-172c638bb21e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a48a3de2cdec4911bc802036e5393302'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:26.033583 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-c0f3ae01-0976-4200-a9f0-172c638bb21e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a48a3de2cdec4911bc802036e5393302'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:26.064712 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-c0f3ae01-0976-4200-a9f0-172c638bb21e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a48a3de2cdec4911bc802036e5393302'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:26.065998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-5e519326-a991-40a9-8e43-8a78f14ee912 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0fa254f4eb59487a91bd5807aca4028a', 'request_id': 'req-5e519326-a991-40a9-8e43-8a78f14ee912'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:26.099937 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-5e519326-a991-40a9-8e43-8a78f14ee912 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0fa254f4eb59487a91bd5807aca4028a', 'request_id': 'req-5e519326-a991-40a9-8e43-8a78f14ee912'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:26.122328 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-c0f3ae01-0976-4200-a9f0-172c638bb21e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a48a3de2cdec4911bc802036e5393302'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:26.130330 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-5e519326-a991-40a9-8e43-8a78f14ee912 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0fa254f4eb59487a91bd5807aca4028a', 'request_id': 'req-5e519326-a991-40a9-8e43-8a78f14ee912'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:26.219305 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 890/1754] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:39:25 2022] DELETE /identity/v3/users/a48a3de2cdec4911bc802036e5393302 => generated 0 bytes in 784 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:26.223892 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f3ae01-0976-4200-a9f0-172c638bb21e admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:26.226089 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-c0f3ae01-0976-4200-a9f0-172c638bb21e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:26.286593 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-5e519326-a991-40a9-8e43-8a78f14ee912 admin admin] The token cache is being invalidate because project 0fa254f4eb59487a91bd5807aca4028a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:39:26.288632 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-5e519326-a991-40a9-8e43-8a78f14ee912 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:26.292191 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-5e519326-a991-40a9-8e43-8a78f14ee912 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:26.306742 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 865/1755] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:39:25 2022] DELETE /identity/v3/projects/0fa254f4eb59487a91bd5807aca4028a => generated 0 bytes in 918 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:26.319847 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e519326-a991-40a9-8e43-8a78f14ee912 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:26.335205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-5e519326-a991-40a9-8e43-8a78f14ee912 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:26.617707 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:26.666209 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f3ae01-0976-4200-a9f0-172c638bb21e admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:26.699446 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0ff2a4-ab5d-42be-9463-047bf7c8348f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:26.703447 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0ff2a4-ab5d-42be-9463-047bf7c8348f admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:26.704541 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0ff2a4-ab5d-42be-9463-047bf7c8348f admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:26.704541 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0ff2a4-ab5d-42be-9463-047bf7c8348f admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:26.705884 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0ff2a4-ab5d-42be-9463-047bf7c8348f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:26.762664 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0ff2a4-ab5d-42be-9463-047bf7c8348f admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:26.822062 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:26.842700 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e519326-a991-40a9-8e43-8a78f14ee912 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:26.862832 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 891/1756] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:26 2022] GET /identity/v3/roles => generated 1420 bytes in 642 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:26.874885 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889671f4-af18-4cf5-9ac1-24753c74a426 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:26.876486 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ffdf59-8e94-459f-953d-600ae4a60607 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:26.877692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889671f4-af18-4cf5-9ac1-24753c74a426 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:26.878420 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889671f4-af18-4cf5-9ac1-24753c74a426 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:26.883790 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ffdf59-8e94-459f-953d-600ae4a60607 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:26.885618 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ffdf59-8e94-459f-953d-600ae4a60607 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:26.894585 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ffdf59-8e94-459f-953d-600ae4a60607 admin admin] PATH_INFO: `/v3/users/b2c5221ac1a8421db778e360fe8868d8` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:26.899565 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ffdf59-8e94-459f-953d-600ae4a60607 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b2c5221ac1a8421db778e360fe8868d8)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:26.913977 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ffdf59-8e94-459f-953d-600ae4a60607 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:27.077160 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-889671f4-af18-4cf5-9ac1-24753c74a426 None None] MFA Rules not processed for user `3ac0d5d8de4943a3888abee3b24b8a6f`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:27.079601 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-889671f4-af18-4cf5-9ac1-24753c74a426 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:27.196526 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-70ffdf59-8e94-459f-953d-600ae4a60607 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b2c5221ac1a8421db778e360fe8868d8'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:27.253747 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 892/1757] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Jul 12 17:39:26 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 389 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:27.255413 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889671f4-af18-4cf5-9ac1-24753c74a426 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:27.263320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-889671f4-af18-4cf5-9ac1-24753c74a426 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:27.294240 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889671f4-af18-4cf5-9ac1-24753c74a426 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:27.300743 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31cc9ff5-6989-4d88-b288-a9a38348178e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:27.301696 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31cc9ff5-6989-4d88-b288-a9a38348178e admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:27.302514 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31cc9ff5-6989-4d88-b288-a9a38348178e admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:27.302967 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31cc9ff5-6989-4d88-b288-a9a38348178e admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:27.304120 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31cc9ff5-6989-4d88-b288-a9a38348178e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:27.305908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31cc9ff5-6989-4d88-b288-a9a38348178e admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:27.322800 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 893/1758] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:27 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:27.326743 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31cc9ff5-6989-4d88-b288-a9a38348178e admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:27.339536 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31cc9ff5-6989-4d88-b288-a9a38348178e admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:27.346366 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f0faef-7dbd-495b-9fc7-79c5f4137acf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:27.347594 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f0faef-7dbd-495b-9fc7-79c5f4137acf admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:27.348050 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f0faef-7dbd-495b-9fc7-79c5f4137acf admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:27.348473 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f0faef-7dbd-495b-9fc7-79c5f4137acf admin admin] PATH_INFO: `/v3/projects/5d5351eda1624eedbc9756965275e71c/users/f2ede09c4d394afe884a2247660b058a/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:27.356311 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-70ffdf59-8e94-459f-953d-600ae4a60607 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b2c5221ac1a8421db778e360fe8868d8'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:27.357107 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-70ffdf59-8e94-459f-953d-600ae4a60607 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b2c5221ac1a8421db778e360fe8868d8'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:27.367767 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-70ffdf59-8e94-459f-953d-600ae4a60607 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b2c5221ac1a8421db778e360fe8868d8'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:27.376320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f0faef-7dbd-495b-9fc7-79c5f4137acf admin admin] RBAC: Authorizing `identity:create_grant(project_id=5d5351eda1624eedbc9756965275e71c, user_id=f2ede09c4d394afe884a2247660b058a, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:27.380710 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f0faef-7dbd-495b-9fc7-79c5f4137acf admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:27.401477 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-70ffdf59-8e94-459f-953d-600ae4a60607 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b2c5221ac1a8421db778e360fe8868d8'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:27.449643 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 866/1759] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:39:26 2022] DELETE /identity/v3/users/b2c5221ac1a8421db778e360fe8868d8 => generated 0 bytes in 1136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:27.450801 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ffdf59-8e94-459f-953d-600ae4a60607 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:27.457716 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-70ffdf59-8e94-459f-953d-600ae4a60607 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:27.558223 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 894/1760] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:39:27 2022] PUT /identity/v3/projects/5d5351eda1624eedbc9756965275e71c/users/f2ede09c4d394afe884a2247660b058a/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:39:27.570149 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429da6ee-e75b-4731-a704-d6f657ef4596 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:27.570776 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429da6ee-e75b-4731-a704-d6f657ef4596 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:27.571348 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429da6ee-e75b-4731-a704-d6f657ef4596 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:27.633712 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ffdf59-8e94-459f-953d-600ae4a60607 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:27.690027 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-429da6ee-e75b-4731-a704-d6f657ef4596 None None] MFA Rules not processed for user `f2ede09c4d394afe884a2247660b058a`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:27.692029 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-429da6ee-e75b-4731-a704-d6f657ef4596 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:27.820097 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816469f9-780d-4629-9705-98217dcf34dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:27.821064 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816469f9-780d-4629-9705-98217dcf34dd admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:27.821395 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816469f9-780d-4629-9705-98217dcf34dd admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:27.821784 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816469f9-780d-4629-9705-98217dcf34dd admin admin] PATH_INFO: `/v3/projects/0b2661b3862b49629d3ef4002215cd49` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:27.824159 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816469f9-780d-4629-9705-98217dcf34dd admin admin] RBAC: Authorizing `identity:delete_project(project_id=0b2661b3862b49629d3ef4002215cd49)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:27.826118 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816469f9-780d-4629-9705-98217dcf34dd admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:27.883854 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 895/1761] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:27 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 311 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:39:27.883854 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-429da6ee-e75b-4731-a704-d6f657ef4596 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:27.883854 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-429da6ee-e75b-4731-a704-d6f657ef4596 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:28.080380 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-816469f9-780d-4629-9705-98217dcf34dd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0b2661b3862b49629d3ef4002215cd49', 'request_id': 'req-816469f9-780d-4629-9705-98217dcf34dd'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:28.095041 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-816469f9-780d-4629-9705-98217dcf34dd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0b2661b3862b49629d3ef4002215cd49', 'request_id': 'req-816469f9-780d-4629-9705-98217dcf34dd'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:28.109559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-816469f9-780d-4629-9705-98217dcf34dd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0b2661b3862b49629d3ef4002215cd49', 'request_id': 'req-816469f9-780d-4629-9705-98217dcf34dd'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:28.194923 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-429da6ee-e75b-4731-a704-d6f657ef4596 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:28.224703 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-816469f9-780d-4629-9705-98217dcf34dd admin admin] The token cache is being invalidate because project 0b2661b3862b49629d3ef4002215cd49 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:39:28.225581 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-816469f9-780d-4629-9705-98217dcf34dd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:28.229528 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-816469f9-780d-4629-9705-98217dcf34dd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:28.231607 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 867/1762] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:39:27 2022] DELETE /identity/v3/projects/0b2661b3862b49629d3ef4002215cd49 => generated 0 bytes in 781 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:28.232275 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816469f9-780d-4629-9705-98217dcf34dd admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:28.235291 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-816469f9-780d-4629-9705-98217dcf34dd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:28.339773 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4faea150-bb15-4e36-b16b-33a9675a5099 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7906d1a19c642bd93507a27bf3e3dae', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:28.340493 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4faea150-bb15-4e36-b16b-33a9675a5099 service swift] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:28.340919 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4faea150-bb15-4e36-b16b-33a9675a5099 service swift] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:28.341220 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4faea150-bb15-4e36-b16b-33a9675a5099 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:28.348983 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-4faea150-bb15-4e36-b16b-33a9675a5099 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:28.461864 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816469f9-780d-4629-9705-98217dcf34dd admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:28.486386 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2924c22-8aeb-4d7f-869b-d4523a59ddb9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:28.487088 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2924c22-8aeb-4d7f-869b-d4523a59ddb9 service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:28.487385 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2924c22-8aeb-4d7f-869b-d4523a59ddb9 service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:28.487668 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2924c22-8aeb-4d7f-869b-d4523a59ddb9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:28.494286 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-e2924c22-8aeb-4d7f-869b-d4523a59ddb9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:28.555545 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4faea150-bb15-4e36-b16b-33a9675a5099 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:28.570631 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4faea150-bb15-4e36-b16b-33a9675a5099 service swift] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:28.597374 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2924c22-8aeb-4d7f-869b-d4523a59ddb9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:28.599489 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2924c22-8aeb-4d7f-869b-d4523a59ddb9 service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:28.614058 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 896/1763] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Jul 12 17:39:27 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 733 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 12 17:39:28.615519 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78d8b2e-b94a-4f85-92f5-226e70728517 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:28.616336 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78d8b2e-b94a-4f85-92f5-226e70728517 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:28.616336 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78d8b2e-b94a-4f85-92f5-226e70728517 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:28.641896 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 868/1764] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:28 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 407 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:39:28.648115 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-b78d8b2e-b94a-4f85-92f5-226e70728517 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:28.667858 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-b78d8b2e-b94a-4f85-92f5-226e70728517 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:28.711717 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2924c22-8aeb-4d7f-869b-d4523a59ddb9 service cinder] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:28.713250 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-e2924c22-8aeb-4d7f-869b-d4523a59ddb9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:28.963243 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2924c22-8aeb-4d7f-869b-d4523a59ddb9 service cinder] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:28.963485 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 897/1765] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 349 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:28.977002 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265de017-9805-4035-9032-dcfe60845e8e None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:28.977002 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265de017-9805-4035-9032-dcfe60845e8e None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:28.977002 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265de017-9805-4035-9032-dcfe60845e8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:28.985211 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d865e7-d25a-4848-9baa-ea358f7a9150 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:28.986129 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d865e7-d25a-4848-9baa-ea358f7a9150 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:28.986455 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d865e7-d25a-4848-9baa-ea358f7a9150 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:28.986744 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d865e7-d25a-4848-9baa-ea358f7a9150 admin admin] PATH_INFO: `/v3/projects/b6fd3357973949308ec8a6ffb78dd938` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:28.989721 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d865e7-d25a-4848-9baa-ea358f7a9150 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b6fd3357973949308ec8a6ffb78dd938)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:28.995939 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d865e7-d25a-4848-9baa-ea358f7a9150 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:29.058783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-265de017-9805-4035-9032-dcfe60845e8e None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:29.060085 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-265de017-9805-4035-9032-dcfe60845e8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:29.120684 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 898/1766] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:39:29.122241 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bd8330-a675-4645-bfbc-4a07b208e4f6 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:29.122567 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bd8330-a675-4645-bfbc-4a07b208e4f6 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:29.122854 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bd8330-a675-4645-bfbc-4a07b208e4f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:29.199869 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-b0bd8330-a675-4645-bfbc-4a07b208e4f6 None None] MFA Rules not processed for user `c8441a4a72154ee3bbb95af66abf2ffd`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:29.204612 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-b0bd8330-a675-4645-bfbc-4a07b208e4f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:29.241826 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:29.346848 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-07d865e7-d25a-4848-9baa-ea358f7a9150 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b6fd3357973949308ec8a6ffb78dd938', 'request_id': 'req-07d865e7-d25a-4848-9baa-ea358f7a9150'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:29.378124 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-07d865e7-d25a-4848-9baa-ea358f7a9150 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b6fd3357973949308ec8a6ffb78dd938', 'request_id': 'req-07d865e7-d25a-4848-9baa-ea358f7a9150'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:29.403926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-07d865e7-d25a-4848-9baa-ea358f7a9150 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b6fd3357973949308ec8a6ffb78dd938', 'request_id': 'req-07d865e7-d25a-4848-9baa-ea358f7a9150'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:29.669476 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 899/1767] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:29 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 305 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:29.669476 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0bd8330-a675-4645-bfbc-4a07b208e4f6 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:29.669476 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0bd8330-a675-4645-bfbc-4a07b208e4f6 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:29.669476 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a0d01c-68e1-46f8-b38d-0195495f89fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:29.672221 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a0d01c-68e1-46f8-b38d-0195495f89fe admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:29.672221 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a0d01c-68e1-46f8-b38d-0195495f89fe admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:29.672221 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a0d01c-68e1-46f8-b38d-0195495f89fe admin admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:29.672221 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a0d01c-68e1-46f8-b38d-0195495f89fe admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:39:29.672221 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a0d01c-68e1-46f8-b38d-0195495f89fe admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:29.672221 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a0d01c-68e1-46f8-b38d-0195495f89fe admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:29.673792 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 900/1768] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:39:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:39:29.673792 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a0d01c-68e1-46f8-b38d-0195495f89fe admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:29.673792 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-d9a0d01c-68e1-46f8-b38d-0195495f89fe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:29.673792 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a0d01c-68e1-46f8-b38d-0195495f89fe admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:29.675444 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8b3c30-b5d8-44ef-aa9a-6f9d6bf975b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:29.675444 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8b3c30-b5d8-44ef-aa9a-6f9d6bf975b2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:29.675444 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8b3c30-b5d8-44ef-aa9a-6f9d6bf975b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:29.675444 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8b3c30-b5d8-44ef-aa9a-6f9d6bf975b2 admin admin] PATH_INFO: `/v3/projects/0a2e89491fcb49dca79518c2d50a71c7` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:29.676363 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8b3c30-b5d8-44ef-aa9a-6f9d6bf975b2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0a2e89491fcb49dca79518c2d50a71c7)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:29.676363 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8b3c30-b5d8-44ef-aa9a-6f9d6bf975b2 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:29.951894 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-07d865e7-d25a-4848-9baa-ea358f7a9150 admin admin] The token cache is being invalidate because project b6fd3357973949308ec8a6ffb78dd938 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:39:29.951894 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-07d865e7-d25a-4848-9baa-ea358f7a9150 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:29.954340 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-07d865e7-d25a-4848-9baa-ea358f7a9150 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:29.956777 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 869/1769] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:39:28 2022] DELETE /identity/v3/projects/b6fd3357973949308ec8a6ffb78dd938 => generated 0 bytes in 1246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:39:29.957812 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d865e7-d25a-4848-9baa-ea358f7a9150 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:29.966553 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-07d865e7-d25a-4848-9baa-ea358f7a9150 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:29.994505 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d865e7-d25a-4848-9baa-ea358f7a9150 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:30.010127 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:30.022048 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-2f8b3c30-b5d8-44ef-aa9a-6f9d6bf975b2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0a2e89491fcb49dca79518c2d50a71c7', 'request_id': 'req-2f8b3c30-b5d8-44ef-aa9a-6f9d6bf975b2'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:30.038588 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-2f8b3c30-b5d8-44ef-aa9a-6f9d6bf975b2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0a2e89491fcb49dca79518c2d50a71c7', 'request_id': 'req-2f8b3c30-b5d8-44ef-aa9a-6f9d6bf975b2'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:30.145060 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6c755c-985a-45a7-ae87-0a4d23f8f6ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:30.147923 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6c755c-985a-45a7-ae87-0a4d23f8f6ed admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:30.148690 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6c755c-985a-45a7-ae87-0a4d23f8f6ed admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:30.149161 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6c755c-985a-45a7-ae87-0a4d23f8f6ed admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:30.150296 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c6c755c-985a-45a7-ae87-0a4d23f8f6ed admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:30.153422 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c6c755c-985a-45a7-ae87-0a4d23f8f6ed admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:30.186375 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-2f8b3c30-b5d8-44ef-aa9a-6f9d6bf975b2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0a2e89491fcb49dca79518c2d50a71c7', 'request_id': 'req-2f8b3c30-b5d8-44ef-aa9a-6f9d6bf975b2'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:30.270971 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-2f8b3c30-b5d8-44ef-aa9a-6f9d6bf975b2 admin admin] The token cache is being invalidate because project 0a2e89491fcb49dca79518c2d50a71c7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:39:30.283306 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-2f8b3c30-b5d8-44ef-aa9a-6f9d6bf975b2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:30.284489 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-2f8b3c30-b5d8-44ef-aa9a-6f9d6bf975b2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:30.294389 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 901/1770] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:39:29 2022] DELETE /identity/v3/projects/0a2e89491fcb49dca79518c2d50a71c7 => generated 0 bytes in 679 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:30.304228 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d346e5-a07f-4244-9154-39f25ccdd08d None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:30.304785 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d346e5-a07f-4244-9154-39f25ccdd08d None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:30.305758 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d346e5-a07f-4244-9154-39f25ccdd08d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:30.382347 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 870/1771] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:39:29 2022] POST /identity/v3/projects => generated 383 bytes in 425 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:30.385095 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75f965f-8a5f-4870-8ec1-af513a41fd8f None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:30.393479 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75f965f-8a5f-4870-8ec1-af513a41fd8f None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:30.402825 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75f965f-8a5f-4870-8ec1-af513a41fd8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:30.417849 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-f4d346e5-a07f-4244-9154-39f25ccdd08d None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:30.419235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-f4d346e5-a07f-4244-9154-39f25ccdd08d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:30.471730 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-d75f965f-8a5f-4870-8ec1-af513a41fd8f None None] MFA Rules not processed for user `9feeaf257bb441b7a14b57f1f493d08f`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:30.473433 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-d75f965f-8a5f-4870-8ec1-af513a41fd8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:30.749365 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 902/1772] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 454 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:30.759614 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333dffcd-a75a-46ed-b39f-9c7e32e6b50a None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:30.763778 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333dffcd-a75a-46ed-b39f-9c7e32e6b50a None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:30.765935 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333dffcd-a75a-46ed-b39f-9c7e32e6b50a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:30.779784 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:30.798082 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 871/1773] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:30 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 415 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:30.800296 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75f965f-8a5f-4870-8ec1-af513a41fd8f None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:30.804915 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-d75f965f-8a5f-4870-8ec1-af513a41fd8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:30.823045 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-333dffcd-a75a-46ed-b39f-9c7e32e6b50a None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:30.823271 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-333dffcd-a75a-46ed-b39f-9c7e32e6b50a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:30.843766 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75f965f-8a5f-4870-8ec1-af513a41fd8f None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:30.854595 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 903/1774] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:30.860260 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f43a09c-0421-4215-87e5-9e9d5b6a4732 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:30.860919 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f43a09c-0421-4215-87e5-9e9d5b6a4732 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:30.865022 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5084937-25a7-478b-bd4b-2cd6c12551ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:30.865318 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f43a09c-0421-4215-87e5-9e9d5b6a4732 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:30.865416 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5084937-25a7-478b-bd4b-2cd6c12551ad admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:30.865416 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5084937-25a7-478b-bd4b-2cd6c12551ad admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:30.867648 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5084937-25a7-478b-bd4b-2cd6c12551ad admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:30.870511 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5084937-25a7-478b-bd4b-2cd6c12551ad admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:30.874490 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5084937-25a7-478b-bd4b-2cd6c12551ad admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:30.906334 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-8f43a09c-0421-4215-87e5-9e9d5b6a4732 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:30.908614 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-8f43a09c-0421-4215-87e5-9e9d5b6a4732 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:30.928447 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 904/1775] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 70 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:30.928447 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861bba45-160f-4821-908f-55034f5495e4 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:30.929565 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861bba45-160f-4821-908f-55034f5495e4 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:30.930452 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861bba45-160f-4821-908f-55034f5495e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:30.968882 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-861bba45-160f-4821-908f-55034f5495e4 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:30.970616 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-861bba45-160f-4821-908f-55034f5495e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:30.985090 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 905/1776] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:30.988196 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d86904-4ce8-4b34-b03f-f477e2499ca5 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:30.988786 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d86904-4ce8-4b34-b03f-f477e2499ca5 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:30.990273 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d86904-4ce8-4b34-b03f-f477e2499ca5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:31.024849 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 872/1777] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:39:30 2022] POST /identity/v3/users => generated 292 bytes in 225 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:31.026484 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5084937-25a7-478b-bd4b-2cd6c12551ad admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:31.055255 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5084937-25a7-478b-bd4b-2cd6c12551ad admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:31.057086 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-11d86904-4ce8-4b34-b03f-f477e2499ca5 None None] MFA Rules not processed for user `bf5bf02ae13d4d86a1b157369ef07a00`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:31.060265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-11d86904-4ce8-4b34-b03f-f477e2499ca5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:31.070088 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca7e131-fbb0-4d6c-bdf4-f2deb1518be5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:31.072845 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca7e131-fbb0-4d6c-bdf4-f2deb1518be5 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:31.075632 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca7e131-fbb0-4d6c-bdf4-f2deb1518be5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:31.075632 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca7e131-fbb0-4d6c-bdf4-f2deb1518be5 admin admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:31.079107 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca7e131-fbb0-4d6c-bdf4-f2deb1518be5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:39:31.081327 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca7e131-fbb0-4d6c-bdf4-f2deb1518be5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:31.084301 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca7e131-fbb0-4d6c-bdf4-f2deb1518be5 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:31.131756 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 873/1778] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:39:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:39:31.131973 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca7e131-fbb0-4d6c-bdf4-f2deb1518be5 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:31.171285 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca7e131-fbb0-4d6c-bdf4-f2deb1518be5 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:31.188495 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1bcb815-a6c5-43d6-9454-df2cf4549be6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:31.190711 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bcb815-a6c5-43d6-9454-df2cf4549be6 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:31.192076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bcb815-a6c5-43d6-9454-df2cf4549be6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:31.192527 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1bcb815-a6c5-43d6-9454-df2cf4549be6 admin admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:31.195067 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1bcb815-a6c5-43d6-9454-df2cf4549be6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:39:31.197087 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1bcb815-a6c5-43d6-9454-df2cf4549be6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:31.207125 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1bcb815-a6c5-43d6-9454-df2cf4549be6 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:31.243216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 874/1780] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:39:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:39:31.243408 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 906/1780] 10.222.0.36 () {58 vars in 1015 bytes} [Tue Jul 12 17:39:30 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 253 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:31.243514 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1bcb815-a6c5-43d6-9454-df2cf4549be6 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:31.243620 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d86904-4ce8-4b34-b03f-f477e2499ca5 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:31.265406 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d86904-4ce8-4b34-b03f-f477e2499ca5 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:31.269233 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1bcb815-a6c5-43d6-9454-df2cf4549be6 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:31.273660 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783c5b9f-af99-4c61-9005-36f4f4ff36c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:31.274217 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783c5b9f-af99-4c61-9005-36f4f4ff36c9 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:31.274665 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783c5b9f-af99-4c61-9005-36f4f4ff36c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:31.275087 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783c5b9f-af99-4c61-9005-36f4f4ff36c9 admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:31.276125 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-783c5b9f-af99-4c61-9005-36f4f4ff36c9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:31.277656 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10429968-231a-4f88-9354-d5a097559a3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:31.279033 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-783c5b9f-af99-4c61-9005-36f4f4ff36c9 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:31.290370 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10429968-231a-4f88-9354-d5a097559a3d admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:31.291887 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10429968-231a-4f88-9354-d5a097559a3d admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:31.292583 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10429968-231a-4f88-9354-d5a097559a3d admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:31.295235 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10429968-231a-4f88-9354-d5a097559a3d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:31.300663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10429968-231a-4f88-9354-d5a097559a3d admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:31.338819 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 875/1781] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:31 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:31.340140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10429968-231a-4f88-9354-d5a097559a3d admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:31.357965 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10429968-231a-4f88-9354-d5a097559a3d admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:31.371287 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3895dbb-8362-47a1-9da1-307d2046ef91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:31.372430 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3895dbb-8362-47a1-9da1-307d2046ef91 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:31.372875 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3895dbb-8362-47a1-9da1-307d2046ef91 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:31.373268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3895dbb-8362-47a1-9da1-307d2046ef91 admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:31.374249 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3895dbb-8362-47a1-9da1-307d2046ef91 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:31.382309 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3895dbb-8362-47a1-9da1-307d2046ef91 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:31.798671 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 907/1782] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:39:31 2022] POST /identity/v3/projects => generated 401 bytes in 557 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:31.812440 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783c5b9f-af99-4c61-9005-36f4f4ff36c9 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:31.816457 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-783c5b9f-af99-4c61-9005-36f4f4ff36c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:31.888169 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:31.915767 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 876/1783] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:39:31 2022] POST /identity/v3/projects => generated 373 bytes in 572 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:31.917796 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3895dbb-8362-47a1-9da1-307d2046ef91 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:32.056247 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783c5b9f-af99-4c61-9005-36f4f4ff36c9 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:32.097519 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:32.168731 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3895dbb-8362-47a1-9da1-307d2046ef91 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:32.181260 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1b047e-2b69-4be1-805e-a856c2e26fa0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:32.182162 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1b047e-2b69-4be1-805e-a856c2e26fa0 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:32.182709 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1b047e-2b69-4be1-805e-a856c2e26fa0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:32.183208 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1b047e-2b69-4be1-805e-a856c2e26fa0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:32.189382 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1b047e-2b69-4be1-805e-a856c2e26fa0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:32.193176 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1b047e-2b69-4be1-805e-a856c2e26fa0 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:32.209398 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 877/1784] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:31 2022] GET /identity/v3/roles => generated 1420 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:32.210883 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1b047e-2b69-4be1-805e-a856c2e26fa0 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:32.239519 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6777b85b-aacf-449b-b16a-4f3ef2ba04ca service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:32.240130 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1b047e-2b69-4be1-805e-a856c2e26fa0 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:32.241103 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6777b85b-aacf-449b-b16a-4f3ef2ba04ca service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:32.241658 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6777b85b-aacf-449b-b16a-4f3ef2ba04ca service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:32.242282 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6777b85b-aacf-449b-b16a-4f3ef2ba04ca service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:32.252606 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5dcbc2-f207-4b4a-9e6d-35ae2c2bf65d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:32.255097 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5dcbc2-f207-4b4a-9e6d-35ae2c2bf65d admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:32.255466 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5dcbc2-f207-4b4a-9e6d-35ae2c2bf65d admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:32.255767 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5dcbc2-f207-4b4a-9e6d-35ae2c2bf65d admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:32.258252 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb5dcbc2-f207-4b4a-9e6d-35ae2c2bf65d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:32.268106 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6777b85b-aacf-449b-b16a-4f3ef2ba04ca service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:32.272287 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6777b85b-aacf-449b-b16a-4f3ef2ba04ca service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:32.279259 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb5dcbc2-f207-4b4a-9e6d-35ae2c2bf65d admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:32.306388 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 908/1785] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:31 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 508 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:39:32.313134 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6777b85b-aacf-449b-b16a-4f3ef2ba04ca service neutron] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:32.349535 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6777b85b-aacf-449b-b16a-4f3ef2ba04ca service neutron] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:32.376287 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cfa66b9-94f1-4fa4-9ccb-99c31b981cb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:32.377012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfa66b9-94f1-4fa4-9ccb-99c31b981cb8 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:32.377365 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfa66b9-94f1-4fa4-9ccb-99c31b981cb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:32.377697 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfa66b9-94f1-4fa4-9ccb-99c31b981cb8 admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:32.382540 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cfa66b9-94f1-4fa4-9ccb-99c31b981cb8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:32.392957 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cfa66b9-94f1-4fa4-9ccb-99c31b981cb8 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:32.412058 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 878/1786] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:39:32 2022] POST /identity/v3/users => generated 301 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:32.412058 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5dcbc2-f207-4b4a-9e6d-35ae2c2bf65d admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:32.450425 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5dcbc2-f207-4b4a-9e6d-35ae2c2bf65d admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:32.472408 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eebc9130-a89f-4c09-a4ee-f0904282b49b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:32.474032 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eebc9130-a89f-4c09-a4ee-f0904282b49b admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:32.487249 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eebc9130-a89f-4c09-a4ee-f0904282b49b admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:32.489608 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eebc9130-a89f-4c09-a4ee-f0904282b49b admin admin] PATH_INFO: `/v3/projects/799000ba12cd423da446691cabee2179/users/3dddcbe59c4447b6b8dc6f059c172c20/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:32.549570 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eebc9130-a89f-4c09-a4ee-f0904282b49b admin admin] RBAC: Authorizing `identity:create_grant(project_id=799000ba12cd423da446691cabee2179, user_id=3dddcbe59c4447b6b8dc6f059c172c20, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:32.561838 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eebc9130-a89f-4c09-a4ee-f0904282b49b admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:32.592069 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 909/1787] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:39:32 2022] POST /identity/v3/users => generated 287 bytes in 285 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:32.594678 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cfa66b9-94f1-4fa4-9ccb-99c31b981cb8 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:32.629960 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cfa66b9-94f1-4fa4-9ccb-99c31b981cb8 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:32.645963 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8431ea6f-c521-4842-bf95-2b1d7e8a383f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:32.646692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8431ea6f-c521-4842-bf95-2b1d7e8a383f admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:32.647015 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8431ea6f-c521-4842-bf95-2b1d7e8a383f admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:32.647284 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8431ea6f-c521-4842-bf95-2b1d7e8a383f admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:32.650048 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8431ea6f-c521-4842-bf95-2b1d7e8a383f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:32.653973 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8431ea6f-c521-4842-bf95-2b1d7e8a383f admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:32.743482 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 910/1788] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:32 2022] GET /identity/v3/roles => generated 1420 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:32.744156 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 879/1789] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:39:32 2022] PUT /identity/v3/projects/799000ba12cd423da446691cabee2179/users/3dddcbe59c4447b6b8dc6f059c172c20/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 334 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:32.756842 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eebc9130-a89f-4c09-a4ee-f0904282b49b admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:32.760749 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8431ea6f-c521-4842-bf95-2b1d7e8a383f admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:32.790451 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8431ea6f-c521-4842-bf95-2b1d7e8a383f admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:33.006767 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d66b6c-09fd-47e4-9d80-4162afd573a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:33.009140 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d66b6c-09fd-47e4-9d80-4162afd573a1 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:33.009140 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d66b6c-09fd-47e4-9d80-4162afd573a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:33.009140 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d66b6c-09fd-47e4-9d80-4162afd573a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:33.009140 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d66b6c-09fd-47e4-9d80-4162afd573a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:33.010034 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eebc9130-a89f-4c09-a4ee-f0904282b49b admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:33.018419 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d66b6c-09fd-47e4-9d80-4162afd573a1 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:33.031102 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384443e4-42f8-414c-83b7-c09a1fdc564c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:33.034298 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384443e4-42f8-414c-83b7-c09a1fdc564c admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:33.034672 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384443e4-42f8-414c-83b7-c09a1fdc564c admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:33.034942 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384443e4-42f8-414c-83b7-c09a1fdc564c admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:33.038101 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-384443e4-42f8-414c-83b7-c09a1fdc564c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:33.044393 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-384443e4-42f8-414c-83b7-c09a1fdc564c admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:33.069505 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 911/1790] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:32 2022] GET /identity/v3/roles => generated 1420 bytes in 325 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:33.080070 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8824b945-fcd5-47d9-b22d-a95a22f133a3 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:33.080070 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8824b945-fcd5-47d9-b22d-a95a22f133a3 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:33.080070 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8824b945-fcd5-47d9-b22d-a95a22f133a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:33.095816 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 880/1791] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:32 2022] GET /identity/v3/roles => generated 1420 bytes in 351 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:33.099115 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384443e4-42f8-414c-83b7-c09a1fdc564c admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:33.144849 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384443e4-42f8-414c-83b7-c09a1fdc564c admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:33.169469 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee3a2a8-1e8e-4314-b5e5-978a55c8ecdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:33.175805 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee3a2a8-1e8e-4314-b5e5-978a55c8ecdc admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:33.177533 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee3a2a8-1e8e-4314-b5e5-978a55c8ecdc admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:33.179214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee3a2a8-1e8e-4314-b5e5-978a55c8ecdc admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:33.180132 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-8824b945-fcd5-47d9-b22d-a95a22f133a3 None None] MFA Rules not processed for user `3dddcbe59c4447b6b8dc6f059c172c20`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:33.181470 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-8824b945-fcd5-47d9-b22d-a95a22f133a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:33.187462 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee3a2a8-1e8e-4314-b5e5-978a55c8ecdc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:33.191405 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee3a2a8-1e8e-4314-b5e5-978a55c8ecdc admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:33.256442 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 881/1792] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:33 2022] GET /identity/v3/roles => generated 1420 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:33.258577 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee3a2a8-1e8e-4314-b5e5-978a55c8ecdc admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:33.349027 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee3a2a8-1e8e-4314-b5e5-978a55c8ecdc admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:33.380557 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d659eb-c1a5-48bc-8abe-dcf383f5875c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:33.381501 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d659eb-c1a5-48bc-8abe-dcf383f5875c admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:33.381804 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d659eb-c1a5-48bc-8abe-dcf383f5875c admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:33.382090 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d659eb-c1a5-48bc-8abe-dcf383f5875c admin admin] PATH_INFO: `/v3/projects/5f5ce208f2b940acaa1b7e2cdf94af74/users/0b24473ec4b543da91ff9be53378bfd5/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:33.403406 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 912/1793] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:33 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 333 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:33.407112 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8824b945-fcd5-47d9-b22d-a95a22f133a3 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:33.450823 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d659eb-c1a5-48bc-8abe-dcf383f5875c admin admin] RBAC: Authorizing `identity:create_grant(project_id=5f5ce208f2b940acaa1b7e2cdf94af74, user_id=0b24473ec4b543da91ff9be53378bfd5, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:33.459911 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d659eb-c1a5-48bc-8abe-dcf383f5875c admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:33.484733 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8824b945-fcd5-47d9-b22d-a95a22f133a3 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:33.503858 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47d8e87-b820-41d4-bc29-96297c617b9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:33.506500 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47d8e87-b820-41d4-bc29-96297c617b9f admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:33.508950 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47d8e87-b820-41d4-bc29-96297c617b9f admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:33.512095 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47d8e87-b820-41d4-bc29-96297c617b9f admin admin] PATH_INFO: `/v3/projects/a224f7b691294e7f89e6b5d8ee401624/users/0a3fc03652ed4c4089b5fb61c088ce29/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:33.546548 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f47d8e87-b820-41d4-bc29-96297c617b9f admin admin] RBAC: Authorizing `identity:create_grant(project_id=a224f7b691294e7f89e6b5d8ee401624, user_id=0a3fc03652ed4c4089b5fb61c088ce29, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:33.548080 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f47d8e87-b820-41d4-bc29-96297c617b9f admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:33.579897 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 882/1794] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:39:33 2022] PUT /identity/v3/projects/5f5ce208f2b940acaa1b7e2cdf94af74/users/0b24473ec4b543da91ff9be53378bfd5/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 322 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:33.594204 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d659eb-c1a5-48bc-8abe-dcf383f5875c admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:33.681592 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 913/1795] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:39:33 2022] PUT /identity/v3/projects/a224f7b691294e7f89e6b5d8ee401624/users/0a3fc03652ed4c4089b5fb61c088ce29/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:33.683748 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df31e41f-063d-4103-bf25-659a00d479a8 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:33.686042 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df31e41f-063d-4103-bf25-659a00d479a8 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:33.688484 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df31e41f-063d-4103-bf25-659a00d479a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:33.795850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:33.824612 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-df31e41f-063d-4103-bf25-659a00d479a8 None None] MFA Rules not processed for user `0b24473ec4b543da91ff9be53378bfd5`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:33.826891 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-df31e41f-063d-4103-bf25-659a00d479a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:33.832346 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:33.850317 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d659eb-c1a5-48bc-8abe-dcf383f5875c admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:34.111644 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b91a64-d2b7-4f1c-aa57-23a8af00c5f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:34.112406 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b91a64-d2b7-4f1c-aa57-23a8af00c5f1 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:34.112958 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b91a64-d2b7-4f1c-aa57-23a8af00c5f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:34.113254 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b91a64-d2b7-4f1c-aa57-23a8af00c5f1 admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:34.114267 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b91a64-d2b7-4f1c-aa57-23a8af00c5f1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:34.116298 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b91a64-d2b7-4f1c-aa57-23a8af00c5f1 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:34.153451 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 914/1796] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:33 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 464 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:34.153451 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520138ca-8008-4249-b3bb-695b1d63da0a None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:34.153451 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520138ca-8008-4249-b3bb-695b1d63da0a None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:34.153451 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520138ca-8008-4249-b3bb-695b1d63da0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:34.251883 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-520138ca-8008-4249-b3bb-695b1d63da0a None None] MFA Rules not processed for user `0a3fc03652ed4c4089b5fb61c088ce29`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:34.253682 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-520138ca-8008-4249-b3bb-695b1d63da0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:34.279124 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 883/1797] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:39:33 2022] POST /identity/v3/projects => generated 385 bytes in 685 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:34.280398 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b91a64-d2b7-4f1c-aa57-23a8af00c5f1 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:34.515642 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:34.539716 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 915/1798] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:34 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 393 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:34.550225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520138ca-8008-4249-b3bb-695b1d63da0a None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:34.590337 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520138ca-8008-4249-b3bb-695b1d63da0a None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:34.593079 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b91a64-d2b7-4f1c-aa57-23a8af00c5f1 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:34.614960 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b77456-9d49-48af-9b27-bdb49b2d0560 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:34.615679 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b77456-9d49-48af-9b27-bdb49b2d0560 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:34.615972 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b77456-9d49-48af-9b27-bdb49b2d0560 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:34.616281 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b77456-9d49-48af-9b27-bdb49b2d0560 admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:34.616985 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b77456-9d49-48af-9b27-bdb49b2d0560 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:34.629716 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b77456-9d49-48af-9b27-bdb49b2d0560 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:34.669483 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543e7f7c-df7e-4805-9891-9eae348b3c55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:34.671468 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543e7f7c-df7e-4805-9891-9eae348b3c55 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:34.674124 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543e7f7c-df7e-4805-9891-9eae348b3c55 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:34.674591 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543e7f7c-df7e-4805-9891-9eae348b3c55 admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:34.678824 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543e7f7c-df7e-4805-9891-9eae348b3c55 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:34.681253 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543e7f7c-df7e-4805-9891-9eae348b3c55 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:34.781506 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 916/1799] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:39:34 2022] POST /identity/v3/users => generated 299 bytes in 241 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:34.782419 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b77456-9d49-48af-9b27-bdb49b2d0560 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:34.812649 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b77456-9d49-48af-9b27-bdb49b2d0560 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:34.835345 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69e9fb6-7196-4082-865c-db2964b180b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:34.836390 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69e9fb6-7196-4082-865c-db2964b180b5 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:34.836886 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69e9fb6-7196-4082-865c-db2964b180b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:34.837389 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69e9fb6-7196-4082-865c-db2964b180b5 admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:34.838307 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c69e9fb6-7196-4082-865c-db2964b180b5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:34.841100 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c69e9fb6-7196-4082-865c-db2964b180b5 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:34.868834 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 884/1800] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:39:34 2022] POST /identity/v3/projects => generated 399 bytes in 589 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:34.879692 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543e7f7c-df7e-4805-9891-9eae348b3c55 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:34.965643 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:34.992902 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 917/1801] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:39:34 2022] POST /identity/v3/projects => generated 371 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:34.998757 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69e9fb6-7196-4082-865c-db2964b180b5 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:35.092099 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:35.200716 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543e7f7c-df7e-4805-9891-9eae348b3c55 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:35.257767 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69e9fb6-7196-4082-865c-db2964b180b5 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:35.277751 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb36e19-93a2-4fae-b559-c388aba70c3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:35.278535 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb36e19-93a2-4fae-b559-c388aba70c3a admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:35.278831 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb36e19-93a2-4fae-b559-c388aba70c3a admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:35.279114 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb36e19-93a2-4fae-b559-c388aba70c3a admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:35.282422 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb36e19-93a2-4fae-b559-c388aba70c3a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:35.292448 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb36e19-93a2-4fae-b559-c388aba70c3a admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:35.413516 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 918/1802] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:39:34 2022] POST /identity/v3/users => generated 306 bytes in 417 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:35.413516 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb36e19-93a2-4fae-b559-c388aba70c3a admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:35.430217 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbba3da-d4e4-436a-be22-1ab40d0bc342 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:35.431817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbba3da-d4e4-436a-be22-1ab40d0bc342 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:35.437543 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbba3da-d4e4-436a-be22-1ab40d0bc342 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:35.437543 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbba3da-d4e4-436a-be22-1ab40d0bc342 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:35.438219 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fbba3da-d4e4-436a-be22-1ab40d0bc342 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:35.444056 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb36e19-93a2-4fae-b559-c388aba70c3a admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:35.446341 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fbba3da-d4e4-436a-be22-1ab40d0bc342 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:35.464411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85adbe9-92b2-4ee4-9ee2-10d1c55575cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:35.465552 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85adbe9-92b2-4ee4-9ee2-10d1c55575cb admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:35.465552 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85adbe9-92b2-4ee4-9ee2-10d1c55575cb admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:35.465727 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85adbe9-92b2-4ee4-9ee2-10d1c55575cb admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:35.467268 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85adbe9-92b2-4ee4-9ee2-10d1c55575cb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:35.474696 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85adbe9-92b2-4ee4-9ee2-10d1c55575cb admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:35.476671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 885/1803] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:34 2022] GET /identity/v3/roles => generated 1420 bytes in 607 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:35.478169 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbba3da-d4e4-436a-be22-1ab40d0bc342 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:35.499751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbba3da-d4e4-436a-be22-1ab40d0bc342 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:35.505220 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b23cf428-d6ed-4f50-817e-b81b3e8d7a69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:35.506129 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23cf428-d6ed-4f50-817e-b81b3e8d7a69 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:35.506293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23cf428-d6ed-4f50-817e-b81b3e8d7a69 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:35.506709 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23cf428-d6ed-4f50-817e-b81b3e8d7a69 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:35.507664 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b23cf428-d6ed-4f50-817e-b81b3e8d7a69 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:35.509509 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b23cf428-d6ed-4f50-817e-b81b3e8d7a69 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:35.526274 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 886/1804] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:35 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:35.527316 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b23cf428-d6ed-4f50-817e-b81b3e8d7a69 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:35.541377 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b23cf428-d6ed-4f50-817e-b81b3e8d7a69 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:35.548560 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ab53ac-c163-4245-a87c-c8e9f75264de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:35.549482 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ab53ac-c163-4245-a87c-c8e9f75264de admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:35.549785 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ab53ac-c163-4245-a87c-c8e9f75264de admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:35.550145 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ab53ac-c163-4245-a87c-c8e9f75264de admin admin] PATH_INFO: `/v3/projects/8e65382389634a17b2e5c127c8456432/users/fb00e58c2a0e46caa2558db9d63c9eae/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:35.578681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 919/1805] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:39:35 2022] POST /identity/v3/users => generated 292 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:35.579624 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85adbe9-92b2-4ee4-9ee2-10d1c55575cb admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:35.586922 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ab53ac-c163-4245-a87c-c8e9f75264de admin admin] RBAC: Authorizing `identity:create_grant(project_id=8e65382389634a17b2e5c127c8456432, user_id=fb00e58c2a0e46caa2558db9d63c9eae, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:35.590401 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ab53ac-c163-4245-a87c-c8e9f75264de admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:35.626853 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85adbe9-92b2-4ee4-9ee2-10d1c55575cb admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:35.643119 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d619fe-37b9-4566-ba65-a30441b21ab3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:35.644811 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d619fe-37b9-4566-ba65-a30441b21ab3 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:35.649621 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d619fe-37b9-4566-ba65-a30441b21ab3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:35.651162 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d619fe-37b9-4566-ba65-a30441b21ab3 admin admin] PATH_INFO: `/v3/projects/fcf615b5ef1e4cf9bdfb7227f38f431d/users/323eb434bb6d44e99d709a68d12cca25/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:35.677442 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 887/1806] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:39:35 2022] PUT /identity/v3/projects/8e65382389634a17b2e5c127c8456432/users/fb00e58c2a0e46caa2558db9d63c9eae/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:35.679505 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ab53ac-c163-4245-a87c-c8e9f75264de admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:35.695792 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d619fe-37b9-4566-ba65-a30441b21ab3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fcf615b5ef1e4cf9bdfb7227f38f431d, user_id=323eb434bb6d44e99d709a68d12cca25, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:35.706933 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d619fe-37b9-4566-ba65-a30441b21ab3 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:35.865210 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 920/1807] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:39:35 2022] PUT /identity/v3/projects/fcf615b5ef1e4cf9bdfb7227f38f431d/users/323eb434bb6d44e99d709a68d12cca25/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 286 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:35.869248 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d619fe-37b9-4566-ba65-a30441b21ab3 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:35.900254 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ab53ac-c163-4245-a87c-c8e9f75264de admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:36.085479 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54feb90c-4842-466f-98f2-4e1fc399bb84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:36.086406 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54feb90c-4842-466f-98f2-4e1fc399bb84 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:36.087142 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54feb90c-4842-466f-98f2-4e1fc399bb84 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:36.087591 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54feb90c-4842-466f-98f2-4e1fc399bb84 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:36.088661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54feb90c-4842-466f-98f2-4e1fc399bb84 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:36.091446 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54feb90c-4842-466f-98f2-4e1fc399bb84 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:36.133314 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 888/1808] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:35 2022] GET /identity/v3/roles => generated 1420 bytes in 443 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:36.145207 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54feb90c-4842-466f-98f2-4e1fc399bb84 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:36.145207 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-54feb90c-4842-466f-98f2-4e1fc399bb84 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:36.158239 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d619fe-37b9-4566-ba65-a30441b21ab3 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:36.158478 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54feb90c-4842-466f-98f2-4e1fc399bb84 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:36.168041 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=94,keystone:DELETE=4,keystone:INSERT=2 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:36.175541 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae548d1-b584-4bb0-9954-e8ad530bf68f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:36.176398 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae548d1-b584-4bb0-9954-e8ad530bf68f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:36.176694 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae548d1-b584-4bb0-9954-e8ad530bf68f admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:36.176975 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae548d1-b584-4bb0-9954-e8ad530bf68f admin admin] PATH_INFO: `/v3/users/f2ede09c4d394afe884a2247660b058a` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:36.178725 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae548d1-b584-4bb0-9954-e8ad530bf68f admin admin] RBAC: Authorizing `identity:delete_user(user_id=f2ede09c4d394afe884a2247660b058a)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:36.181174 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae548d1-b584-4bb0-9954-e8ad530bf68f admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:36.181764 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a99bac-6e75-4da8-8446-e5c1812e5d67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:36.182496 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a99bac-6e75-4da8-8446-e5c1812e5d67 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:36.182784 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a99bac-6e75-4da8-8446-e5c1812e5d67 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:36.183062 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a99bac-6e75-4da8-8446-e5c1812e5d67 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:36.183818 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6a99bac-6e75-4da8-8446-e5c1812e5d67 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:36.185446 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6a99bac-6e75-4da8-8446-e5c1812e5d67 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:36.208329 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 921/1809] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:35 2022] GET /identity/v3/roles => generated 1420 bytes in 340 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:36.209639 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a99bac-6e75-4da8-8446-e5c1812e5d67 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:36.228028 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a99bac-6e75-4da8-8446-e5c1812e5d67 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:36.238352 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0179472e-7a2d-4008-b83f-80a921149345 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:36.238865 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0179472e-7a2d-4008-b83f-80a921149345 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:36.239345 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0179472e-7a2d-4008-b83f-80a921149345 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:36.239725 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0179472e-7a2d-4008-b83f-80a921149345 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:36.241226 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0179472e-7a2d-4008-b83f-80a921149345 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:36.243010 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0179472e-7a2d-4008-b83f-80a921149345 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:36.306856 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 922/1810] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:36 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:36.308818 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54e86a6-b049-4658-a027-fc88987b61b0 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:36.309238 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54e86a6-b049-4658-a027-fc88987b61b0 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:36.309646 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54e86a6-b049-4658-a027-fc88987b61b0 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:36.315865 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 923/1811] 10.222.0.16 () {54 vars in 921 bytes} [Tue Jul 12 17:39:36 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:39:36.316963 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54e86a6-b049-4658-a027-fc88987b61b0 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:36.348522 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54e86a6-b049-4658-a027-fc88987b61b0 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:36.354294 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f58556a-4dc6-417d-8aac-045f6452fd81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:36.355383 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f58556a-4dc6-417d-8aac-045f6452fd81 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:36.356064 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f58556a-4dc6-417d-8aac-045f6452fd81 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:36.356461 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f58556a-4dc6-417d-8aac-045f6452fd81 admin admin] PATH_INFO: `/v3/projects/fb0146591fe543e383d4c5aa20109336/users/d3bd428cb2a34f9683c7d7acb5195f36/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:36.385317 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f58556a-4dc6-417d-8aac-045f6452fd81 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fb0146591fe543e383d4c5aa20109336, user_id=d3bd428cb2a34f9683c7d7acb5195f36, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:36.388625 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f58556a-4dc6-417d-8aac-045f6452fd81 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:36.394442 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-cae548d1-b584-4bb0-9954-e8ad530bf68f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f2ede09c4d394afe884a2247660b058a'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:36.531772 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-cae548d1-b584-4bb0-9954-e8ad530bf68f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f2ede09c4d394afe884a2247660b058a'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:36.534776 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-cae548d1-b584-4bb0-9954-e8ad530bf68f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f2ede09c4d394afe884a2247660b058a'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:36.540206 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 924/1812] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:39:36 2022] PUT /identity/v3/projects/fb0146591fe543e383d4c5aa20109336/users/d3bd428cb2a34f9683c7d7acb5195f36/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:36.540922 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f58556a-4dc6-417d-8aac-045f6452fd81 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:36.549376 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-9f58556a-4dc6-417d-8aac-045f6452fd81 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:36.558202 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-cae548d1-b584-4bb0-9954-e8ad530bf68f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f2ede09c4d394afe884a2247660b058a'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:36.614961 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-cae548d1-b584-4bb0-9954-e8ad530bf68f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f2ede09c4d394afe884a2247660b058a'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:36.623508 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:36.636695 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 889/1813] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:39:36 2022] DELETE /identity/v3/users/f2ede09c4d394afe884a2247660b058a => generated 0 bytes in 500 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:36.636695 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae548d1-b584-4bb0-9954-e8ad530bf68f admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:36.636695 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-cae548d1-b584-4bb0-9954-e8ad530bf68f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:36.699964 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f58556a-4dc6-417d-8aac-045f6452fd81 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:36.809439 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae548d1-b584-4bb0-9954-e8ad530bf68f admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:36.817838 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4697dd76-3131-4620-889a-af7b614e5924 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:36.818887 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4697dd76-3131-4620-889a-af7b614e5924 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:36.819185 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4697dd76-3131-4620-889a-af7b614e5924 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:36.819470 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4697dd76-3131-4620-889a-af7b614e5924 admin admin] PATH_INFO: `/v3/projects/fcf615b5ef1e4cf9bdfb7227f38f431d/users/323eb434bb6d44e99d709a68d12cca25/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:36.821981 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4697dd76-3131-4620-889a-af7b614e5924 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fcf615b5ef1e4cf9bdfb7227f38f431d, user_id=323eb434bb6d44e99d709a68d12cca25, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:36.825750 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4697dd76-3131-4620-889a-af7b614e5924 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:36.862560 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 890/1814] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:39:36 2022] PUT /identity/v3/projects/fcf615b5ef1e4cf9bdfb7227f38f431d/users/323eb434bb6d44e99d709a68d12cca25/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:36.864326 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10a6a6f-ed3a-4920-a99b-bcb0275c8dd6 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:36.865704 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10a6a6f-ed3a-4920-a99b-bcb0275c8dd6 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:36.865997 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10a6a6f-ed3a-4920-a99b-bcb0275c8dd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:36.892446 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f04cb1-f0e9-462f-a03e-c0d8cdb1b5c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:36.896222 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f04cb1-f0e9-462f-a03e-c0d8cdb1b5c0 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:36.896570 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f04cb1-f0e9-462f-a03e-c0d8cdb1b5c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:36.896860 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f04cb1-f0e9-462f-a03e-c0d8cdb1b5c0 admin admin] PATH_INFO: `/v3/projects/8e65382389634a17b2e5c127c8456432/users/fb00e58c2a0e46caa2558db9d63c9eae/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:36.899826 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-c10a6a6f-ed3a-4920-a99b-bcb0275c8dd6 None None] MFA Rules not processed for user `9012d5b88f554dc596582a7693c2a72a`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:36.901208 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-c10a6a6f-ed3a-4920-a99b-bcb0275c8dd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:36.905003 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71f04cb1-f0e9-462f-a03e-c0d8cdb1b5c0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8e65382389634a17b2e5c127c8456432, user_id=fb00e58c2a0e46caa2558db9d63c9eae, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:36.912469 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71f04cb1-f0e9-462f-a03e-c0d8cdb1b5c0 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:36.959860 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 925/1815] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:39:36 2022] PUT /identity/v3/projects/8e65382389634a17b2e5c127c8456432/users/fb00e58c2a0e46caa2558db9d63c9eae/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 419 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:36.961078 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f04cb1-f0e9-462f-a03e-c0d8cdb1b5c0 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:36.977739 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-71f04cb1-f0e9-462f-a03e-c0d8cdb1b5c0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:36.983214 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:37.147180 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f04cb1-f0e9-462f-a03e-c0d8cdb1b5c0 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:37.150952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 891/1816] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Jul 12 17:39:36 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 288 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:37.156982 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6f057b-ab9d-406a-a750-fca135f7ae21 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:37.158383 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6f057b-ab9d-406a-a750-fca135f7ae21 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:37.160159 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6f057b-ab9d-406a-a750-fca135f7ae21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:37.162383 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a177e8-7929-452d-b8e5-5717b515a4af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:37.169235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a177e8-7929-452d-b8e5-5717b515a4af admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:37.169235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a177e8-7929-452d-b8e5-5717b515a4af admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:37.169572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a177e8-7929-452d-b8e5-5717b515a4af admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:37.171597 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38a177e8-7929-452d-b8e5-5717b515a4af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:37.178097 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38a177e8-7929-452d-b8e5-5717b515a4af admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:37.215528 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:37.218877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 926/1817] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:36 2022] GET /identity/v3/roles => generated 1420 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:37.226897 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-1e6f057b-ab9d-406a-a750-fca135f7ae21 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:37.228716 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9a856a-2e16-474b-9a97-9a5890566215 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:37.229116 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9a856a-2e16-474b-9a97-9a5890566215 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:37.229317 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-1e6f057b-ab9d-406a-a750-fca135f7ae21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:37.229499 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9a856a-2e16-474b-9a97-9a5890566215 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:37.246596 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 892/1818] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:37.248573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b706dbb5-2dd2-4dc9-9b04-321b7c8f25b0 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:37.262979 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b706dbb5-2dd2-4dc9-9b04-321b7c8f25b0 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:37.264308 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b706dbb5-2dd2-4dc9-9b04-321b7c8f25b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:37.323237 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-5e9a856a-2e16-474b-9a97-9a5890566215 None None] MFA Rules not processed for user `323eb434bb6d44e99d709a68d12cca25`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:37.324570 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-5e9a856a-2e16-474b-9a97-9a5890566215 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:37.336248 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-b706dbb5-2dd2-4dc9-9b04-321b7c8f25b0 None None] MFA Rules not processed for user `fb00e58c2a0e46caa2558db9d63c9eae`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:37.338206 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-b706dbb5-2dd2-4dc9-9b04-321b7c8f25b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:37.452783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:37.517393 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 927/1819] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:37 2022] POST /identity/v3/auth/tokens => generated 3333 bytes in 295 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:37.517393 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9a856a-2e16-474b-9a97-9a5890566215 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:37.517662 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 893/1820] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:37 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 270 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:37.531299 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b706dbb5-2dd2-4dc9-9b04-321b7c8f25b0 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:37.534366 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-b706dbb5-2dd2-4dc9-9b04-321b7c8f25b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:37.557301 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9a856a-2e16-474b-9a97-9a5890566215 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:37.585992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f88a74-c1d7-43ee-b749-636e21e6800e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:37.585992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f88a74-c1d7-43ee-b749-636e21e6800e admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:37.585992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f88a74-c1d7-43ee-b749-636e21e6800e admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:37.593788 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f88a74-c1d7-43ee-b749-636e21e6800e admin admin] PATH_INFO: `/v3/projects/fb0146591fe543e383d4c5aa20109336/users/d3bd428cb2a34f9683c7d7acb5195f36/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:37.601778 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f88a74-c1d7-43ee-b749-636e21e6800e admin admin] RBAC: Authorizing `identity:create_grant(project_id=fb0146591fe543e383d4c5aa20109336, user_id=d3bd428cb2a34f9683c7d7acb5195f36, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:37.605463 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f88a74-c1d7-43ee-b749-636e21e6800e admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:37.636236 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 928/1821] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:39:37 2022] PUT /identity/v3/projects/fb0146591fe543e383d4c5aa20109336/users/d3bd428cb2a34f9683c7d7acb5195f36/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:37.636236 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f88a74-c1d7-43ee-b749-636e21e6800e admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:37.644456 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-d9f88a74-c1d7-43ee-b749-636e21e6800e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:37.726691 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b706dbb5-2dd2-4dc9-9b04-321b7c8f25b0 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:37.851665 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f88a74-c1d7-43ee-b749-636e21e6800e admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:37.869525 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34cbb28c-8442-4fe1-83e5-212a0da168d0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:37.870510 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34cbb28c-8442-4fe1-83e5-212a0da168d0 service nova] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:37.870956 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34cbb28c-8442-4fe1-83e5-212a0da168d0 service nova] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:37.871426 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34cbb28c-8442-4fe1-83e5-212a0da168d0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:37.880566 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8832fa15-fb4f-4f99-be6f-a114b6d33f8a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:37.881546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8832fa15-fb4f-4f99-be6f-a114b6d33f8a service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:37.882400 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8832fa15-fb4f-4f99-be6f-a114b6d33f8a service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:37.882400 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8832fa15-fb4f-4f99-be6f-a114b6d33f8a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:37.893186 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34cbb28c-8442-4fe1-83e5-212a0da168d0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:37.896791 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34cbb28c-8442-4fe1-83e5-212a0da168d0 service nova] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:37.921257 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 929/1822] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:37 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:39:37.932664 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34cbb28c-8442-4fe1-83e5-212a0da168d0 service nova] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:37.943365 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-34cbb28c-8442-4fe1-83e5-212a0da168d0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:37.985184 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:38.018878 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:38.055306 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8832fa15-fb4f-4f99-be6f-a114b6d33f8a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:38.056801 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8832fa15-fb4f-4f99-be6f-a114b6d33f8a service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:38.097475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 894/1823] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:37 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 580 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:39:38.098648 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8832fa15-fb4f-4f99-be6f-a114b6d33f8a service neutron] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:38.108828 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-8832fa15-fb4f-4f99-be6f-a114b6d33f8a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:38.137939 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34cbb28c-8442-4fe1-83e5-212a0da168d0 service nova] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:38.142648 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f5ed273-eeb5-4fe5-8471-9356c628666e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:38.143375 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5ed273-eeb5-4fe5-8471-9356c628666e service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:38.143676 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5ed273-eeb5-4fe5-8471-9356c628666e service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:38.144011 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f5ed273-eeb5-4fe5-8471-9356c628666e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:38.154289 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8832fa15-fb4f-4f99-be6f-a114b6d33f8a service neutron] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:38.167144 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdfc1553-5f6a-4808-b014-e4d80fcefd47 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:38.171339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfc1553-5f6a-4808-b014-e4d80fcefd47 service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:38.171339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfc1553-5f6a-4808-b014-e4d80fcefd47 service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:38.171339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfc1553-5f6a-4808-b014-e4d80fcefd47 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:38.219321 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:38.276603 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:38.297430 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5ed273-eeb5-4fe5-8471-9356c628666e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:38.300094 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f5ed273-eeb5-4fe5-8471-9356c628666e service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:38.305315 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdfc1553-5f6a-4808-b014-e4d80fcefd47 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:38.306895 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdfc1553-5f6a-4808-b014-e4d80fcefd47 service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:38.324133 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 930/1825] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:37 2022] GET /identity/v3/auth/tokens => generated 3333 bytes in 392 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:39:38.325630 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f984ab5-7792-4891-979e-c53839369a40 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:38.325902 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f984ab5-7792-4891-979e-c53839369a40 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:38.326187 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f984ab5-7792-4891-979e-c53839369a40 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:38.336167 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 931/1826] 10.222.0.16 () {54 vars in 921 bytes} [Tue Jul 12 17:39:38 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:39:38.336167 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c267fc9-5921-4e8f-996c-ee8fec050d36 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:38.336167 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c267fc9-5921-4e8f-996c-ee8fec050d36 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:38.336167 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c267fc9-5921-4e8f-996c-ee8fec050d36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:38.341378 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 895/1826] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:38 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 225 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:39:38.383802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-3c267fc9-5921-4e8f-996c-ee8fec050d36 None None] MFA Rules not processed for user `d3bd428cb2a34f9683c7d7acb5195f36`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:38.385418 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-3c267fc9-5921-4e8f-996c-ee8fec050d36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:38.421856 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36fc428f-89c1-495a-b949-61b76bc11a3a None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:38.422277 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36fc428f-89c1-495a-b949-61b76bc11a3a None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:38.422668 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36fc428f-89c1-495a-b949-61b76bc11a3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:38.523248 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-36fc428f-89c1-495a-b949-61b76bc11a3a None None] MFA Rules not processed for user `9012d5b88f554dc596582a7693c2a72a`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:38.528992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-36fc428f-89c1-495a-b949-61b76bc11a3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:38.574073 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 932/1827] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:38 2022] POST /identity/v3/auth/tokens => generated 3305 bytes in 236 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:38.593400 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 896/1828] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Jul 12 17:39:38 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:39:38.690236 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c267fc9-5921-4e8f-996c-ee8fec050d36 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:38.709242 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c267fc9-5921-4e8f-996c-ee8fec050d36 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:38.717955 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee901e41-76af-453b-beb7-627d6bf83a9d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:38.717955 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee901e41-76af-453b-beb7-627d6bf83a9d service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:38.718155 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36fc428f-89c1-495a-b949-61b76bc11a3a None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:38.718240 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee901e41-76af-453b-beb7-627d6bf83a9d service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:38.718240 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee901e41-76af-453b-beb7-627d6bf83a9d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:38.719366 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-36fc428f-89c1-495a-b949-61b76bc11a3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:38.744183 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:38.758771 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36fc428f-89c1-495a-b949-61b76bc11a3a None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:38.759442 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee901e41-76af-453b-beb7-627d6bf83a9d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:38.762687 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee901e41-76af-453b-beb7-627d6bf83a9d service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:38.773662 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98043725-532d-4f34-b9a3-864d8761162f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:38.774350 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98043725-532d-4f34-b9a3-864d8761162f service nova] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:38.774653 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98043725-532d-4f34-b9a3-864d8761162f service nova] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:38.774930 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98043725-532d-4f34-b9a3-864d8761162f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:38.786957 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:38.798748 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98043725-532d-4f34-b9a3-864d8761162f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:38.805008 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98043725-532d-4f34-b9a3-864d8761162f service nova] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:38.820033 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 933/1829] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:38 2022] GET /identity/v3/auth/tokens => generated 3305 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:39:38.845267 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 897/1830] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:38 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:39:38.906466 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee901e41-76af-453b-beb7-627d6bf83a9d service cinder] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:38.927333 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee901e41-76af-453b-beb7-627d6bf83a9d service cinder] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:38.952729 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f7ed8e-eba2-49c2-82ab-88b2c28a32ea service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:38.953485 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f7ed8e-eba2-49c2-82ab-88b2c28a32ea service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:38.955181 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f7ed8e-eba2-49c2-82ab-88b2c28a32ea service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:38.955604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f7ed8e-eba2-49c2-82ab-88b2c28a32ea service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:38.966081 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-96f7ed8e-eba2-49c2-82ab-88b2c28a32ea service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:39.109866 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f7ed8e-eba2-49c2-82ab-88b2c28a32ea service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:39.111685 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f7ed8e-eba2-49c2-82ab-88b2c28a32ea service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:39.138641 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 934/1831] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:38 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 234 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:39:39.406327 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98043725-532d-4f34-b9a3-864d8761162f service nova] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:39.423820 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98043725-532d-4f34-b9a3-864d8761162f service nova] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:39.443602 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00da3a0d-17b7-453b-b1f0-9e76037c7c8e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:39.444284 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00da3a0d-17b7-453b-b1f0-9e76037c7c8e service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:39.444613 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00da3a0d-17b7-453b-b1f0-9e76037c7c8e service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:39.444897 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00da3a0d-17b7-453b-b1f0-9e76037c7c8e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:39.448203 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-00da3a0d-17b7-453b-b1f0-9e76037c7c8e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:39.498367 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:39.627632 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00da3a0d-17b7-453b-b1f0-9e76037c7c8e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:39.640991 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00da3a0d-17b7-453b-b1f0-9e76037c7c8e service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:39.685852 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 898/1832] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:39 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 280 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:39:41.100021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f7ed8e-eba2-49c2-82ab-88b2c28a32ea service cinder] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:41.102334 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-96f7ed8e-eba2-49c2-82ab-88b2c28a32ea service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:41.143094 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f7ed8e-eba2-49c2-82ab-88b2c28a32ea service cinder] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:41.152688 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea460782-f517-4899-8d6b-d6d0a9ba22ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:41.152688 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea460782-f517-4899-8d6b-d6d0a9ba22ae admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:41.152688 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea460782-f517-4899-8d6b-d6d0a9ba22ae admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:41.153234 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea460782-f517-4899-8d6b-d6d0a9ba22ae admin admin] PATH_INFO: `/v3/projects/5d5351eda1624eedbc9756965275e71c` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:41.170646 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea460782-f517-4899-8d6b-d6d0a9ba22ae admin admin] RBAC: Authorizing `identity:delete_project(project_id=5d5351eda1624eedbc9756965275e71c)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:41.173551 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea460782-f517-4899-8d6b-d6d0a9ba22ae admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:41.314632 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:41.393421 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ea460782-f517-4899-8d6b-d6d0a9ba22ae admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5d5351eda1624eedbc9756965275e71c', 'request_id': 'req-ea460782-f517-4899-8d6b-d6d0a9ba22ae'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:41.447024 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ea460782-f517-4899-8d6b-d6d0a9ba22ae admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5d5351eda1624eedbc9756965275e71c', 'request_id': 'req-ea460782-f517-4899-8d6b-d6d0a9ba22ae'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:41.571734 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ea460782-f517-4899-8d6b-d6d0a9ba22ae admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5d5351eda1624eedbc9756965275e71c', 'request_id': 'req-ea460782-f517-4899-8d6b-d6d0a9ba22ae'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:41.584250 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:41.607233 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ea460782-f517-4899-8d6b-d6d0a9ba22ae admin admin] The token cache is being invalidate because project 5d5351eda1624eedbc9756965275e71c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:39:41.611607 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ea460782-f517-4899-8d6b-d6d0a9ba22ae admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:41.611607 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ea460782-f517-4899-8d6b-d6d0a9ba22ae admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:41.616746 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 935/1833] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:39:41 2022] DELETE /identity/v3/projects/5d5351eda1624eedbc9756965275e71c => generated 0 bytes in 516 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:41.625393 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651d85c3-d755-4419-add7-24716c7f7b37 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:41.625601 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651d85c3-d755-4419-add7-24716c7f7b37 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:41.625881 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651d85c3-d755-4419-add7-24716c7f7b37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:41.666234 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-651d85c3-d755-4419-add7-24716c7f7b37 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:41.669954 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-651d85c3-d755-4419-add7-24716c7f7b37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:41.804660 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 899/1834] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:41.818338 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dcfb515-46ae-4b2f-a7db-3c78607114cf None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:41.818679 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dcfb515-46ae-4b2f-a7db-3c78607114cf None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:41.818965 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dcfb515-46ae-4b2f-a7db-3c78607114cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:41.941385 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-9dcfb515-46ae-4b2f-a7db-3c78607114cf None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:41.944146 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-9dcfb515-46ae-4b2f-a7db-3c78607114cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:41.972133 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 936/1835] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:41.972608 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651d85c3-d755-4419-add7-24716c7f7b37 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:42.005055 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651d85c3-d755-4419-add7-24716c7f7b37 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:42.020322 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860e9848-ba52-4277-bc20-666c63b1f81d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:42.021200 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860e9848-ba52-4277-bc20-666c63b1f81d admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:42.021564 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860e9848-ba52-4277-bc20-666c63b1f81d admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:42.023579 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860e9848-ba52-4277-bc20-666c63b1f81d admin admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:42.024450 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-860e9848-ba52-4277-bc20-666c63b1f81d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:39:42.024773 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-860e9848-ba52-4277-bc20-666c63b1f81d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:42.027487 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-860e9848-ba52-4277-bc20-666c63b1f81d admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:42.069530 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 900/1836] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:39:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:39:42.074884 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dcfb515-46ae-4b2f-a7db-3c78607114cf None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:42.087043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dcfb515-46ae-4b2f-a7db-3c78607114cf None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:42.100950 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b01eca-d973-4c84-8961-06d20a0c312e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:42.101631 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b01eca-d973-4c84-8961-06d20a0c312e admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:42.101918 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b01eca-d973-4c84-8961-06d20a0c312e admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:42.105141 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b01eca-d973-4c84-8961-06d20a0c312e admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:42.105946 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50b01eca-d973-4c84-8961-06d20a0c312e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:42.110389 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50b01eca-d973-4c84-8961-06d20a0c312e admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:42.368729 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 937/1837] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:39:42 2022] POST /identity/v3/projects => generated 385 bytes in 294 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:42.372963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860e9848-ba52-4277-bc20-666c63b1f81d admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:42.390370 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860e9848-ba52-4277-bc20-666c63b1f81d admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:42.647480 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ff3e65-07a8-489f-ace6-3a6ca2f84633 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:42.648208 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ff3e65-07a8-489f-ace6-3a6ca2f84633 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:42.660214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ff3e65-07a8-489f-ace6-3a6ca2f84633 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:42.660595 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ff3e65-07a8-489f-ace6-3a6ca2f84633 admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:42.661388 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ff3e65-07a8-489f-ace6-3a6ca2f84633 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:42.663837 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ff3e65-07a8-489f-ace6-3a6ca2f84633 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:42.831615 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 901/1838] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:39:42 2022] POST /identity/v3/users => generated 293 bytes in 459 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:42.836071 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b01eca-d973-4c84-8961-06d20a0c312e admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:42.849384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b01eca-d973-4c84-8961-06d20a0c312e admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:42.869995 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e2bb0c-fa47-428e-9dd5-fde34d39f9e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:42.872568 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e2bb0c-fa47-428e-9dd5-fde34d39f9e4 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:42.876871 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e2bb0c-fa47-428e-9dd5-fde34d39f9e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:42.883332 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e2bb0c-fa47-428e-9dd5-fde34d39f9e4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:42.883332 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93e2bb0c-fa47-428e-9dd5-fde34d39f9e4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:42.883332 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93e2bb0c-fa47-428e-9dd5-fde34d39f9e4 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:42.948085 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 938/1839] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:42 2022] GET /identity/v3/roles => generated 1420 bytes in 112 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:42.949040 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ff3e65-07a8-489f-ace6-3a6ca2f84633 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:42.978627 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ff3e65-07a8-489f-ace6-3a6ca2f84633 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:42.999733 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b4edd5-2d72-492b-bb79-3513bf8b1d14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:43.000677 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b4edd5-2d72-492b-bb79-3513bf8b1d14 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:43.000981 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b4edd5-2d72-492b-bb79-3513bf8b1d14 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:43.001934 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b4edd5-2d72-492b-bb79-3513bf8b1d14 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:43.003418 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b4edd5-2d72-492b-bb79-3513bf8b1d14 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:43.012884 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b4edd5-2d72-492b-bb79-3513bf8b1d14 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:43.074827 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 902/1840] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:42 2022] GET /identity/v3/roles => generated 1420 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:43.079401 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e2bb0c-fa47-428e-9dd5-fde34d39f9e4 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:43.102772 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e2bb0c-fa47-428e-9dd5-fde34d39f9e4 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:43.117281 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3de69c3-db82-4f3e-975c-76ccf72aafbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:43.119465 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3de69c3-db82-4f3e-975c-76ccf72aafbf admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:43.122304 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3de69c3-db82-4f3e-975c-76ccf72aafbf admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:43.124402 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3de69c3-db82-4f3e-975c-76ccf72aafbf admin admin] PATH_INFO: `/v3/projects/2896afab75b54d8eb3692d56ea153bf5/users/fe88c6cc9f1a4ef4be4f57d93864ac3f/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:43.169256 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3de69c3-db82-4f3e-975c-76ccf72aafbf admin admin] RBAC: Authorizing `identity:create_grant(project_id=2896afab75b54d8eb3692d56ea153bf5, user_id=fe88c6cc9f1a4ef4be4f57d93864ac3f, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:43.169579 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3de69c3-db82-4f3e-975c-76ccf72aafbf admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:43.731532 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 939/1841] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:39:43 2022] PUT /identity/v3/projects/2896afab75b54d8eb3692d56ea153bf5/users/fe88c6cc9f1a4ef4be4f57d93864ac3f/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 643 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:39:43.737338 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e91de2-41d6-4b9b-ae94-97b33d393206 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:43.738477 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e91de2-41d6-4b9b-ae94-97b33d393206 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:43.739487 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e91de2-41d6-4b9b-ae94-97b33d393206 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:43.787422 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-59e91de2-41d6-4b9b-ae94-97b33d393206 None None] MFA Rules not processed for user `fe88c6cc9f1a4ef4be4f57d93864ac3f`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:43.788902 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-59e91de2-41d6-4b9b-ae94-97b33d393206 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:43.796471 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:43.876547 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 903/1842] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:43 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:43.881581 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3de69c3-db82-4f3e-975c-76ccf72aafbf admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:43.961615 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3de69c3-db82-4f3e-975c-76ccf72aafbf admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:43.968410 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90212e72-bc0c-407c-a5c7-1f37c62e4116 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:43.969617 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90212e72-bc0c-407c-a5c7-1f37c62e4116 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:43.969617 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90212e72-bc0c-407c-a5c7-1f37c62e4116 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:43.970057 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90212e72-bc0c-407c-a5c7-1f37c62e4116 admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:43.971500 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90212e72-bc0c-407c-a5c7-1f37c62e4116 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:43.974043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90212e72-bc0c-407c-a5c7-1f37c62e4116 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:44.137685 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 940/1843] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:39:43 2022] POST /identity/v3/projects => generated 385 bytes in 257 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:39:44.139232 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e91de2-41d6-4b9b-ae94-97b33d393206 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:44.160041 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e91de2-41d6-4b9b-ae94-97b33d393206 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:44.288271 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5126035b-9024-4df8-a5ae-5913b52ac45d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:44.289194 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5126035b-9024-4df8-a5ae-5913b52ac45d admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:44.289590 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5126035b-9024-4df8-a5ae-5913b52ac45d admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:44.290286 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5126035b-9024-4df8-a5ae-5913b52ac45d admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:44.291682 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5126035b-9024-4df8-a5ae-5913b52ac45d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:44.294920 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5126035b-9024-4df8-a5ae-5913b52ac45d admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:44.416181 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 904/1844] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:39:44 2022] POST /identity/v3/users => generated 293 bytes in 278 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:44.419885 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90212e72-bc0c-407c-a5c7-1f37c62e4116 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:44.434857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90212e72-bc0c-407c-a5c7-1f37c62e4116 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:44.445327 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6707bf7-8065-4443-8c21-6a0995f00f6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:44.446156 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6707bf7-8065-4443-8c21-6a0995f00f6c admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:44.446476 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6707bf7-8065-4443-8c21-6a0995f00f6c admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:44.446772 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6707bf7-8065-4443-8c21-6a0995f00f6c admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:44.447549 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6707bf7-8065-4443-8c21-6a0995f00f6c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:44.449871 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6707bf7-8065-4443-8c21-6a0995f00f6c admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:44.477919 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 941/1845] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:44 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:44.481890 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5126035b-9024-4df8-a5ae-5913b52ac45d admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:44.503542 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5126035b-9024-4df8-a5ae-5913b52ac45d admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:44.510504 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95bcd6c1-b493-41ab-8b83-1234e8619f6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:44.511322 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bcd6c1-b493-41ab-8b83-1234e8619f6b admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:44.511756 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bcd6c1-b493-41ab-8b83-1234e8619f6b admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:44.512118 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bcd6c1-b493-41ab-8b83-1234e8619f6b admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:44.513729 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95bcd6c1-b493-41ab-8b83-1234e8619f6b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:44.516346 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95bcd6c1-b493-41ab-8b83-1234e8619f6b admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:44.530748 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 905/1846] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:44 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:44.538827 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6707bf7-8065-4443-8c21-6a0995f00f6c admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:44.566134 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6707bf7-8065-4443-8c21-6a0995f00f6c admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:44.572175 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe42e003-ad46-419d-a979-2637e6e66ad4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:44.574460 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe42e003-ad46-419d-a979-2637e6e66ad4 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:44.576359 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe42e003-ad46-419d-a979-2637e6e66ad4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:44.577039 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe42e003-ad46-419d-a979-2637e6e66ad4 admin admin] PATH_INFO: `/v3/projects/18f82954e6a348c9b4d5db65b70a119c/users/b5531745ed794253858009c00754c5a4/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:44.646055 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe42e003-ad46-419d-a979-2637e6e66ad4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=18f82954e6a348c9b4d5db65b70a119c, user_id=b5531745ed794253858009c00754c5a4, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:44.677985 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95bcd6c1-b493-41ab-8b83-1234e8619f6b admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:44.680167 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-95bcd6c1-b493-41ab-8b83-1234e8619f6b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:44.681643 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe42e003-ad46-419d-a979-2637e6e66ad4 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:44.714311 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95bcd6c1-b493-41ab-8b83-1234e8619f6b admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:44.740237 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98b4cd1b-5266-4bd8-b7b0-5c59f8f8e8bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:44.741314 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b4cd1b-5266-4bd8-b7b0-5c59f8f8e8bd admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:44.741642 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b4cd1b-5266-4bd8-b7b0-5c59f8f8e8bd admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:44.741933 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98b4cd1b-5266-4bd8-b7b0-5c59f8f8e8bd admin admin] PATH_INFO: `/v3/users/3dddcbe59c4447b6b8dc6f059c172c20` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:44.743423 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b4cd1b-5266-4bd8-b7b0-5c59f8f8e8bd admin admin] RBAC: Authorizing `identity:delete_user(user_id=3dddcbe59c4447b6b8dc6f059c172c20)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:44.750814 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98b4cd1b-5266-4bd8-b7b0-5c59f8f8e8bd admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:44.801796 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 942/1847] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:39:44 2022] PUT /identity/v3/projects/18f82954e6a348c9b4d5db65b70a119c/users/b5531745ed794253858009c00754c5a4/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:39:44.812992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ac3087-9a85-4c11-8f8d-c5a2da8f1cb7 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:44.813299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ac3087-9a85-4c11-8f8d-c5a2da8f1cb7 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:44.813586 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ac3087-9a85-4c11-8f8d-c5a2da8f1cb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:44.871812 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-c1ac3087-9a85-4c11-8f8d-c5a2da8f1cb7 None None] MFA Rules not processed for user `b5531745ed794253858009c00754c5a4`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:44.874350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-c1ac3087-9a85-4c11-8f8d-c5a2da8f1cb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:44.912433 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:44.967387 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 943/1848] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:44 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:44.972831 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ac3087-9a85-4c11-8f8d-c5a2da8f1cb7 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:45.065586 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-98b4cd1b-5266-4bd8-b7b0-5c59f8f8e8bd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3dddcbe59c4447b6b8dc6f059c172c20'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:45.088700 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ac3087-9a85-4c11-8f8d-c5a2da8f1cb7 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:45.094786 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c842ff-1141-4f21-9b18-644079c45da8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:45.095494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c842ff-1141-4f21-9b18-644079c45da8 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:45.095786 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c842ff-1141-4f21-9b18-644079c45da8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:45.096069 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c842ff-1141-4f21-9b18-644079c45da8 admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:45.096787 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c842ff-1141-4f21-9b18-644079c45da8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:45.099336 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c842ff-1141-4f21-9b18-644079c45da8 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:45.164444 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-98b4cd1b-5266-4bd8-b7b0-5c59f8f8e8bd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3dddcbe59c4447b6b8dc6f059c172c20'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:45.164444 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-98b4cd1b-5266-4bd8-b7b0-5c59f8f8e8bd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3dddcbe59c4447b6b8dc6f059c172c20'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:45.180403 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-98b4cd1b-5266-4bd8-b7b0-5c59f8f8e8bd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3dddcbe59c4447b6b8dc6f059c172c20'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:45.241101 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-98b4cd1b-5266-4bd8-b7b0-5c59f8f8e8bd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3dddcbe59c4447b6b8dc6f059c172c20'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:45.244123 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 944/1849] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:39:44 2022] POST /identity/v3/projects => generated 383 bytes in 272 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:45.245375 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c842ff-1141-4f21-9b18-644079c45da8 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:45.252491 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-81c842ff-1141-4f21-9b18-644079c45da8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:45.282064 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 906/1850] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:39:44 2022] DELETE /identity/v3/users/3dddcbe59c4447b6b8dc6f059c172c20 => generated 0 bytes in 604 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:45.287552 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb44be77-3441-445e-bac5-1ad78d1d1b02 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:45.287812 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb44be77-3441-445e-bac5-1ad78d1d1b02 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:45.288108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb44be77-3441-445e-bac5-1ad78d1d1b02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:45.311630 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-cb44be77-3441-445e-bac5-1ad78d1d1b02 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:45.312894 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-cb44be77-3441-445e-bac5-1ad78d1d1b02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:45.420586 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c842ff-1141-4f21-9b18-644079c45da8 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:45.435683 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 907/1851] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:39:45.447972 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb44be77-3441-445e-bac5-1ad78d1d1b02 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:45.449700 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-cb44be77-3441-445e-bac5-1ad78d1d1b02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:45.497085 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da8a376-561d-4fa7-a294-fd74eb60c027 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:45.497772 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da8a376-561d-4fa7-a294-fd74eb60c027 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:45.498102 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da8a376-561d-4fa7-a294-fd74eb60c027 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:45.498402 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da8a376-561d-4fa7-a294-fd74eb60c027 admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:45.499207 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da8a376-561d-4fa7-a294-fd74eb60c027 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:45.501211 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3da8a376-561d-4fa7-a294-fd74eb60c027 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:45.519284 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb44be77-3441-445e-bac5-1ad78d1d1b02 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:45.523124 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85aa1a62-ead1-4c7b-9f09-e42e95e8aa5d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:45.523791 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85aa1a62-ead1-4c7b-9f09-e42e95e8aa5d service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:45.524085 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85aa1a62-ead1-4c7b-9f09-e42e95e8aa5d service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:45.524376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85aa1a62-ead1-4c7b-9f09-e42e95e8aa5d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:45.537284 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85aa1a62-ead1-4c7b-9f09-e42e95e8aa5d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:45.538789 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85aa1a62-ead1-4c7b-9f09-e42e95e8aa5d service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:45.549331 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 908/1852] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:45 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:39:45.643081 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 945/1853] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:39:45 2022] POST /identity/v3/users => generated 298 bytes in 399 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:45.649998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85aa1a62-ead1-4c7b-9f09-e42e95e8aa5d service neutron] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:45.664781 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85aa1a62-ead1-4c7b-9f09-e42e95e8aa5d service neutron] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:45.681052 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235c0e16-3132-4076-bcd1-282038662f8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:45.681878 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235c0e16-3132-4076-bcd1-282038662f8c admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:45.682176 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235c0e16-3132-4076-bcd1-282038662f8c admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:45.682484 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235c0e16-3132-4076-bcd1-282038662f8c admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:45.683380 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235c0e16-3132-4076-bcd1-282038662f8c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:45.685084 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235c0e16-3132-4076-bcd1-282038662f8c admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:45.713030 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 909/1854] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:45 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:45.715022 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da8a376-561d-4fa7-a294-fd74eb60c027 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:45.730884 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da8a376-561d-4fa7-a294-fd74eb60c027 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:45.740126 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c960e23-f4e6-4ac5-9ef2-91ff491576f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:45.742983 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c960e23-f4e6-4ac5-9ef2-91ff491576f8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:45.742983 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c960e23-f4e6-4ac5-9ef2-91ff491576f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:45.742983 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c960e23-f4e6-4ac5-9ef2-91ff491576f8 admin admin] PATH_INFO: `/v3/projects/8f5604ce279346d39eba7e3db6a61e4f/users/e9eaf24f35a14c3489341eb0ccd0a53d/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:45.764705 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c960e23-f4e6-4ac5-9ef2-91ff491576f8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8f5604ce279346d39eba7e3db6a61e4f, user_id=e9eaf24f35a14c3489341eb0ccd0a53d, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:45.767867 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c960e23-f4e6-4ac5-9ef2-91ff491576f8 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:45.871648 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 946/1855] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:39:45 2022] PUT /identity/v3/projects/8f5604ce279346d39eba7e3db6a61e4f/users/e9eaf24f35a14c3489341eb0ccd0a53d/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:45.877238 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235c0e16-3132-4076-bcd1-282038662f8c admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:45.894647 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235c0e16-3132-4076-bcd1-282038662f8c admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:46.026896 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e639d0c-d20d-447f-b963-36011760bf1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:46.031839 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e639d0c-d20d-447f-b963-36011760bf1f admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:46.031839 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e639d0c-d20d-447f-b963-36011760bf1f admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:46.031839 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:46.031839 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e639d0c-d20d-447f-b963-36011760bf1f admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:46.032249 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e639d0c-d20d-447f-b963-36011760bf1f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:46.033919 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e639d0c-d20d-447f-b963-36011760bf1f admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:46.045507 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 910/1856] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:45 2022] GET /identity/v3/roles => generated 1420 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:46.050537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c960e23-f4e6-4ac5-9ef2-91ff491576f8 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:46.065793 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c960e23-f4e6-4ac5-9ef2-91ff491576f8 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:46.070568 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9da31743-4c54-4ddd-9cee-b84b1fd809d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:46.072010 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da31743-4c54-4ddd-9cee-b84b1fd809d8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:46.072875 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da31743-4c54-4ddd-9cee-b84b1fd809d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:46.073490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da31743-4c54-4ddd-9cee-b84b1fd809d8 admin admin] PATH_INFO: `/v3/projects/8f5604ce279346d39eba7e3db6a61e4f/users/e9eaf24f35a14c3489341eb0ccd0a53d/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:46.075815 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9da31743-4c54-4ddd-9cee-b84b1fd809d8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8f5604ce279346d39eba7e3db6a61e4f, user_id=e9eaf24f35a14c3489341eb0ccd0a53d, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:46.079069 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9da31743-4c54-4ddd-9cee-b84b1fd809d8 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:46.095802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 947/1857] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:39:46 2022] PUT /identity/v3/projects/8f5604ce279346d39eba7e3db6a61e4f/users/e9eaf24f35a14c3489341eb0ccd0a53d/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:39:46.107290 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90bf709-2511-4556-bb7b-7b4405c21269 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:46.108153 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90bf709-2511-4556-bb7b-7b4405c21269 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:46.108857 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90bf709-2511-4556-bb7b-7b4405c21269 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:46.149648 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-a90bf709-2511-4556-bb7b-7b4405c21269 None None] MFA Rules not processed for user `e9eaf24f35a14c3489341eb0ccd0a53d`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:46.151994 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-a90bf709-2511-4556-bb7b-7b4405c21269 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:46.157657 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9da31743-4c54-4ddd-9cee-b84b1fd809d8 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:46.160010 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-9da31743-4c54-4ddd-9cee-b84b1fd809d8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:46.282566 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9da31743-4c54-4ddd-9cee-b84b1fd809d8 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:46.288740 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a95d52f-78a4-4ad8-9c34-09f2a2c0a752 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:46.289828 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a95d52f-78a4-4ad8-9c34-09f2a2c0a752 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:46.290118 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a95d52f-78a4-4ad8-9c34-09f2a2c0a752 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:46.290397 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a95d52f-78a4-4ad8-9c34-09f2a2c0a752 admin admin] PATH_INFO: `/v3/projects/799000ba12cd423da446691cabee2179` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:46.291893 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a95d52f-78a4-4ad8-9c34-09f2a2c0a752 admin admin] RBAC: Authorizing `identity:delete_project(project_id=799000ba12cd423da446691cabee2179)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:46.294533 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a95d52f-78a4-4ad8-9c34-09f2a2c0a752 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:46.329682 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 911/1858] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:46 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 223 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:46.392789 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a90bf709-2511-4556-bb7b-7b4405c21269 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:46.394464 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-a90bf709-2511-4556-bb7b-7b4405c21269 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:46.471331 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-3a95d52f-78a4-4ad8-9c34-09f2a2c0a752 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '799000ba12cd423da446691cabee2179', 'request_id': 'req-3a95d52f-78a4-4ad8-9c34-09f2a2c0a752'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:46.490841 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-3a95d52f-78a4-4ad8-9c34-09f2a2c0a752 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '799000ba12cd423da446691cabee2179', 'request_id': 'req-3a95d52f-78a4-4ad8-9c34-09f2a2c0a752'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:46.511378 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a90bf709-2511-4556-bb7b-7b4405c21269 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:46.623970 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9d93b8-8ce6-4863-b2b6-939391df017e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:46.625907 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9d93b8-8ce6-4863-b2b6-939391df017e service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:46.626452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9d93b8-8ce6-4863-b2b6-939391df017e service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:46.626853 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9d93b8-8ce6-4863-b2b6-939391df017e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:46.629138 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-ef9d93b8-8ce6-4863-b2b6-939391df017e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:46.630273 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-3a95d52f-78a4-4ad8-9c34-09f2a2c0a752 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '799000ba12cd423da446691cabee2179', 'request_id': 'req-3a95d52f-78a4-4ad8-9c34-09f2a2c0a752'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:46.650027 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-3a95d52f-78a4-4ad8-9c34-09f2a2c0a752 admin admin] The token cache is being invalidate because project 799000ba12cd423da446691cabee2179 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:39:46.650408 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-3a95d52f-78a4-4ad8-9c34-09f2a2c0a752 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:46.651344 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-3a95d52f-78a4-4ad8-9c34-09f2a2c0a752 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:46.657807 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 948/1859] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:39:46 2022] DELETE /identity/v3/projects/799000ba12cd423da446691cabee2179 => generated 0 bytes in 497 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:39:46.660318 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a95d52f-78a4-4ad8-9c34-09f2a2c0a752 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:46.663175 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-3a95d52f-78a4-4ad8-9c34-09f2a2c0a752 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:46.773822 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a95d52f-78a4-4ad8-9c34-09f2a2c0a752 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:46.780844 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ccc8af4-2dbb-4dba-9be9-878b4376440e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:46.782124 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccc8af4-2dbb-4dba-9be9-878b4376440e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:46.782647 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccc8af4-2dbb-4dba-9be9-878b4376440e admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:46.783352 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccc8af4-2dbb-4dba-9be9-878b4376440e admin admin] PATH_INFO: `/v3/users/fb00e58c2a0e46caa2558db9d63c9eae` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:46.785006 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ccc8af4-2dbb-4dba-9be9-878b4376440e admin admin] RBAC: Authorizing `identity:delete_user(user_id=fb00e58c2a0e46caa2558db9d63c9eae)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:46.785254 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9d93b8-8ce6-4863-b2b6-939391df017e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:46.787357 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9d93b8-8ce6-4863-b2b6-939391df017e service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:46.787598 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ccc8af4-2dbb-4dba-9be9-878b4376440e admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:46.805669 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 912/1860] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:46 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 413 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:39:47.052093 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-6ccc8af4-2dbb-4dba-9be9-878b4376440e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fb00e58c2a0e46caa2558db9d63c9eae'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:47.052792 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-6ccc8af4-2dbb-4dba-9be9-878b4376440e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fb00e58c2a0e46caa2558db9d63c9eae'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:47.053463 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-6ccc8af4-2dbb-4dba-9be9-878b4376440e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fb00e58c2a0e46caa2558db9d63c9eae'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:47.062227 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-6ccc8af4-2dbb-4dba-9be9-878b4376440e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fb00e58c2a0e46caa2558db9d63c9eae'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:47.079201 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-6ccc8af4-2dbb-4dba-9be9-878b4376440e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fb00e58c2a0e46caa2558db9d63c9eae'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:47.194698 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 949/1861] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:39:46 2022] DELETE /identity/v3/users/fb00e58c2a0e46caa2558db9d63c9eae => generated 0 bytes in 536 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:47.694872 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9d93b8-8ce6-4863-b2b6-939391df017e service cinder] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:47.711906 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9d93b8-8ce6-4863-b2b6-939391df017e service cinder] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:47.757266 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ccc8af4-2dbb-4dba-9be9-878b4376440e admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:47.760384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-6ccc8af4-2dbb-4dba-9be9-878b4376440e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:47.788342 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:47.850312 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16056ff6-8d8e-49d6-8ce3-06f7e06032e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:47.851416 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16056ff6-8d8e-49d6-8ce3-06f7e06032e5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:47.851837 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16056ff6-8d8e-49d6-8ce3-06f7e06032e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:47.852131 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16056ff6-8d8e-49d6-8ce3-06f7e06032e5 admin admin] PATH_INFO: `/v3/projects/8e65382389634a17b2e5c127c8456432` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:47.853625 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16056ff6-8d8e-49d6-8ce3-06f7e06032e5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8e65382389634a17b2e5c127c8456432)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:47.856175 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16056ff6-8d8e-49d6-8ce3-06f7e06032e5 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:47.918367 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ccc8af4-2dbb-4dba-9be9-878b4376440e admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:47.924229 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f01359-3a24-4f21-ad17-f1dabad397bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:47.925130 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f01359-3a24-4f21-ad17-f1dabad397bb admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:47.925416 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f01359-3a24-4f21-ad17-f1dabad397bb admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:47.925693 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f01359-3a24-4f21-ad17-f1dabad397bb admin admin] PATH_INFO: `/v3/users/0b24473ec4b543da91ff9be53378bfd5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:47.927046 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73f01359-3a24-4f21-ad17-f1dabad397bb admin admin] RBAC: Authorizing `identity:delete_user(user_id=0b24473ec4b543da91ff9be53378bfd5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:47.929048 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73f01359-3a24-4f21-ad17-f1dabad397bb admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:47.979261 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-16056ff6-8d8e-49d6-8ce3-06f7e06032e5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8e65382389634a17b2e5c127c8456432', 'request_id': 'req-16056ff6-8d8e-49d6-8ce3-06f7e06032e5'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:47.989161 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-16056ff6-8d8e-49d6-8ce3-06f7e06032e5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8e65382389634a17b2e5c127c8456432', 'request_id': 'req-16056ff6-8d8e-49d6-8ce3-06f7e06032e5'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:47.995331 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-16056ff6-8d8e-49d6-8ce3-06f7e06032e5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8e65382389634a17b2e5c127c8456432', 'request_id': 'req-16056ff6-8d8e-49d6-8ce3-06f7e06032e5'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:48.084254 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-16056ff6-8d8e-49d6-8ce3-06f7e06032e5 admin admin] The token cache is being invalidate because project 8e65382389634a17b2e5c127c8456432 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:39:48.084554 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-16056ff6-8d8e-49d6-8ce3-06f7e06032e5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:48.085201 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-16056ff6-8d8e-49d6-8ce3-06f7e06032e5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:48.086867 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 913/1862] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:39:47 2022] DELETE /identity/v3/projects/8e65382389634a17b2e5c127c8456432 => generated 0 bytes in 393 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:48.096061 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6c3c5c-f37d-43d1-9ec0-04e1a8d3a01b None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:48.096322 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6c3c5c-f37d-43d1-9ec0-04e1a8d3a01b None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:48.096613 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6c3c5c-f37d-43d1-9ec0-04e1a8d3a01b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:48.116417 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-4b6c3c5c-f37d-43d1-9ec0-04e1a8d3a01b None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:48.121563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-4b6c3c5c-f37d-43d1-9ec0-04e1a8d3a01b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:48.175352 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-73f01359-3a24-4f21-ad17-f1dabad397bb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0b24473ec4b543da91ff9be53378bfd5'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:48.176050 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-73f01359-3a24-4f21-ad17-f1dabad397bb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0b24473ec4b543da91ff9be53378bfd5'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:48.176681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-73f01359-3a24-4f21-ad17-f1dabad397bb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0b24473ec4b543da91ff9be53378bfd5'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:48.184741 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-73f01359-3a24-4f21-ad17-f1dabad397bb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0b24473ec4b543da91ff9be53378bfd5'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:48.199932 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-73f01359-3a24-4f21-ad17-f1dabad397bb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0b24473ec4b543da91ff9be53378bfd5'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:48.212744 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 914/1863] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:39:48.214638 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dcf61fc-0a01-41b4-9bfa-acba469cf7f2 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:48.214937 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dcf61fc-0a01-41b4-9bfa-acba469cf7f2 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:48.215222 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dcf61fc-0a01-41b4-9bfa-acba469cf7f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:48.235150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-3dcf61fc-0a01-41b4-9bfa-acba469cf7f2 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:48.236409 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-3dcf61fc-0a01-41b4-9bfa-acba469cf7f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:48.244360 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 915/1864] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:48.249641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dcf61fc-0a01-41b4-9bfa-acba469cf7f2 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:48.258815 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dcf61fc-0a01-41b4-9bfa-acba469cf7f2 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:48.264140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d03d15-3d75-4c78-a8e4-b4100b3b2aec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:48.264872 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d03d15-3d75-4c78-a8e4-b4100b3b2aec admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:48.265148 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d03d15-3d75-4c78-a8e4-b4100b3b2aec admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:48.265430 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d03d15-3d75-4c78-a8e4-b4100b3b2aec admin admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:48.266167 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d03d15-3d75-4c78-a8e4-b4100b3b2aec admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:39:48.266505 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d03d15-3d75-4c78-a8e4-b4100b3b2aec admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:48.268146 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d03d15-3d75-4c78-a8e4-b4100b3b2aec admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:48.281251 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 916/1865] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:39:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:39:48.287604 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d03d15-3d75-4c78-a8e4-b4100b3b2aec admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:48.299469 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d03d15-3d75-4c78-a8e4-b4100b3b2aec admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:48.304190 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76822f75-8277-402e-b772-1ebdb727250d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:48.304882 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76822f75-8277-402e-b772-1ebdb727250d admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:48.305167 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76822f75-8277-402e-b772-1ebdb727250d admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:48.305450 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76822f75-8277-402e-b772-1ebdb727250d admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:48.306147 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76822f75-8277-402e-b772-1ebdb727250d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:48.308096 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76822f75-8277-402e-b772-1ebdb727250d admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:48.313044 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 950/1866] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:39:47 2022] DELETE /identity/v3/users/0b24473ec4b543da91ff9be53378bfd5 => generated 0 bytes in 557 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:48.317740 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd6ccad-54cc-4f95-80bb-e04ada9799b6 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:48.318032 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd6ccad-54cc-4f95-80bb-e04ada9799b6 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:48.318308 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd6ccad-54cc-4f95-80bb-e04ada9799b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:48.335544 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-1dd6ccad-54cc-4f95-80bb-e04ada9799b6 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:48.339579 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-1dd6ccad-54cc-4f95-80bb-e04ada9799b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:48.396768 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 951/1867] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:39:48.407079 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd6ccad-54cc-4f95-80bb-e04ada9799b6 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:48.408696 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-1dd6ccad-54cc-4f95-80bb-e04ada9799b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:48.441563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 917/1868] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:39:48 2022] POST /identity/v3/projects => generated 373 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:48.445982 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76822f75-8277-402e-b772-1ebdb727250d admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:48.472260 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd6ccad-54cc-4f95-80bb-e04ada9799b6 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:48.515032 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76822f75-8277-402e-b772-1ebdb727250d admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:48.524381 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e72e8e5-d634-479e-b536-e18aad2b0145 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:48.525088 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e72e8e5-d634-479e-b536-e18aad2b0145 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:48.525372 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e72e8e5-d634-479e-b536-e18aad2b0145 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:48.525650 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e72e8e5-d634-479e-b536-e18aad2b0145 admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:48.526561 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e72e8e5-d634-479e-b536-e18aad2b0145 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:48.527173 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c31a3210-9983-48ce-852f-65de512e6b85 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:48.527919 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31a3210-9983-48ce-852f-65de512e6b85 service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:48.528325 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31a3210-9983-48ce-852f-65de512e6b85 service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:48.528535 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31a3210-9983-48ce-852f-65de512e6b85 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:48.528659 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e72e8e5-d634-479e-b536-e18aad2b0145 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:48.543247 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c31a3210-9983-48ce-852f-65de512e6b85 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:48.545713 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c31a3210-9983-48ce-852f-65de512e6b85 service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:48.556914 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 952/1869] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:48 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:39:48.674478 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 918/1870] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:39:48 2022] POST /identity/v3/users => generated 287 bytes in 228 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:48.687647 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c31a3210-9983-48ce-852f-65de512e6b85 service neutron] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:48.692122 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c31a3210-9983-48ce-852f-65de512e6b85 service neutron] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:48.698455 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a51775-4ec8-4b2e-ada1-8c6efe152ef2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:48.699220 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a51775-4ec8-4b2e-ada1-8c6efe152ef2 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:48.699624 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a51775-4ec8-4b2e-ada1-8c6efe152ef2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:48.699976 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a51775-4ec8-4b2e-ada1-8c6efe152ef2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:48.700776 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64a51775-4ec8-4b2e-ada1-8c6efe152ef2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:48.702554 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64a51775-4ec8-4b2e-ada1-8c6efe152ef2 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:48.713915 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 953/1871] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:48 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:48.718819 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e72e8e5-d634-479e-b536-e18aad2b0145 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:48.729301 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e72e8e5-d634-479e-b536-e18aad2b0145 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:48.734777 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc270a74-d5da-43c7-8cb4-ce0aedbe10b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:48.735507 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc270a74-d5da-43c7-8cb4-ce0aedbe10b5 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:48.735821 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc270a74-d5da-43c7-8cb4-ce0aedbe10b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:48.736106 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc270a74-d5da-43c7-8cb4-ce0aedbe10b5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:48.736872 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc270a74-d5da-43c7-8cb4-ce0aedbe10b5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:48.738528 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc270a74-d5da-43c7-8cb4-ce0aedbe10b5 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:48.753399 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 919/1872] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:48 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:48.758269 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a51775-4ec8-4b2e-ada1-8c6efe152ef2 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:48.772211 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a51775-4ec8-4b2e-ada1-8c6efe152ef2 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:48.786797 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9ad3b0-a956-49d9-83a0-4daf6e16e55a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:48.788514 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9ad3b0-a956-49d9-83a0-4daf6e16e55a admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:48.789330 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9ad3b0-a956-49d9-83a0-4daf6e16e55a admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:48.789965 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9ad3b0-a956-49d9-83a0-4daf6e16e55a admin admin] PATH_INFO: `/v3/projects/b3cb433fc8c44c87a642aaa9011ff624/users/2c4deca7942842759f795ab5299b44a3/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:48.828062 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e9ad3b0-a956-49d9-83a0-4daf6e16e55a admin admin] RBAC: Authorizing `identity:create_grant(project_id=b3cb433fc8c44c87a642aaa9011ff624, user_id=2c4deca7942842759f795ab5299b44a3, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:48.831345 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e9ad3b0-a956-49d9-83a0-4daf6e16e55a admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:48.950285 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 954/1873] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:39:48 2022] PUT /identity/v3/projects/b3cb433fc8c44c87a642aaa9011ff624/users/2c4deca7942842759f795ab5299b44a3/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:39:48.956199 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e3f2a6-f086-4e50-9acf-137f666c7233 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:48.956480 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e3f2a6-f086-4e50-9acf-137f666c7233 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:48.956777 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e3f2a6-f086-4e50-9acf-137f666c7233 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:48.987988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-37e3f2a6-f086-4e50-9acf-137f666c7233 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:48.989224 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-37e3f2a6-f086-4e50-9acf-137f666c7233 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:49.023172 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9ad3b0-a956-49d9-83a0-4daf6e16e55a admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:49.025847 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-2e9ad3b0-a956-49d9-83a0-4daf6e16e55a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:49.084459 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 920/1874] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:49.096239 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e3f2a6-f086-4e50-9acf-137f666c7233 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:49.097817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-37e3f2a6-f086-4e50-9acf-137f666c7233 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:49.138160 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9ad3b0-a956-49d9-83a0-4daf6e16e55a admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:49.145158 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b639d88b-288b-4dec-9e92-343105ad53cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:49.146960 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b639d88b-288b-4dec-9e92-343105ad53cc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:49.147251 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b639d88b-288b-4dec-9e92-343105ad53cc admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:49.147710 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b639d88b-288b-4dec-9e92-343105ad53cc admin admin] PATH_INFO: `/v3/projects/5f5ce208f2b940acaa1b7e2cdf94af74` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:49.149962 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b639d88b-288b-4dec-9e92-343105ad53cc admin admin] RBAC: Authorizing `identity:delete_project(project_id=5f5ce208f2b940acaa1b7e2cdf94af74)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:49.153718 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b639d88b-288b-4dec-9e92-343105ad53cc admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:49.208583 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e3f2a6-f086-4e50-9acf-137f666c7233 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:49.215314 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec74da5-b4b2-4ffe-acf8-f98dc4cac06e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:49.217225 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec74da5-b4b2-4ffe-acf8-f98dc4cac06e service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:49.217769 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec74da5-b4b2-4ffe-acf8-f98dc4cac06e service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:49.218294 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec74da5-b4b2-4ffe-acf8-f98dc4cac06e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:49.236051 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec74da5-b4b2-4ffe-acf8-f98dc4cac06e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:49.238953 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec74da5-b4b2-4ffe-acf8-f98dc4cac06e service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:49.257166 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 921/1875] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:49 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:39:49.292617 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-b639d88b-288b-4dec-9e92-343105ad53cc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5f5ce208f2b940acaa1b7e2cdf94af74', 'request_id': 'req-b639d88b-288b-4dec-9e92-343105ad53cc'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:49.302264 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-b639d88b-288b-4dec-9e92-343105ad53cc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5f5ce208f2b940acaa1b7e2cdf94af74', 'request_id': 'req-b639d88b-288b-4dec-9e92-343105ad53cc'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:49.376789 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-b639d88b-288b-4dec-9e92-343105ad53cc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5f5ce208f2b940acaa1b7e2cdf94af74', 'request_id': 'req-b639d88b-288b-4dec-9e92-343105ad53cc'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:49.391110 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-b639d88b-288b-4dec-9e92-343105ad53cc admin admin] The token cache is being invalidate because project 5f5ce208f2b940acaa1b7e2cdf94af74 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:39:49.391414 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-b639d88b-288b-4dec-9e92-343105ad53cc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:49.392228 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-b639d88b-288b-4dec-9e92-343105ad53cc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:49.398200 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 955/1876] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:39:49 2022] DELETE /identity/v3/projects/5f5ce208f2b940acaa1b7e2cdf94af74 => generated 0 bytes in 372 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:49.403574 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec74da5-b4b2-4ffe-acf8-f98dc4cac06e service neutron] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:49.418069 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec74da5-b4b2-4ffe-acf8-f98dc4cac06e service neutron] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:49.516940 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60cce2a-cc3d-4cd3-a1b2-86bc482e2d03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:49.518580 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60cce2a-cc3d-4cd3-a1b2-86bc482e2d03 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:49.518580 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60cce2a-cc3d-4cd3-a1b2-86bc482e2d03 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:49.518580 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60cce2a-cc3d-4cd3-a1b2-86bc482e2d03 admin admin] PATH_INFO: `/v3/users/323eb434bb6d44e99d709a68d12cca25` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:49.520097 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60cce2a-cc3d-4cd3-a1b2-86bc482e2d03 admin admin] RBAC: Authorizing `identity:delete_user(user_id=323eb434bb6d44e99d709a68d12cca25)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:49.522326 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60cce2a-cc3d-4cd3-a1b2-86bc482e2d03 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:49.766902 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:49.827954 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-a60cce2a-cc3d-4cd3-a1b2-86bc482e2d03 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '323eb434bb6d44e99d709a68d12cca25'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:49.959480 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-a60cce2a-cc3d-4cd3-a1b2-86bc482e2d03 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '323eb434bb6d44e99d709a68d12cca25'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:49.960305 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-a60cce2a-cc3d-4cd3-a1b2-86bc482e2d03 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '323eb434bb6d44e99d709a68d12cca25'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:49.985385 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-a60cce2a-cc3d-4cd3-a1b2-86bc482e2d03 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '323eb434bb6d44e99d709a68d12cca25'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:49.995786 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:INSERT=4,keystone:SELECT=89,keystone:DELETE=7 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:50.008518 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-a60cce2a-cc3d-4cd3-a1b2-86bc482e2d03 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '323eb434bb6d44e99d709a68d12cca25'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:50.022821 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 922/1877] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:39:49 2022] DELETE /identity/v3/users/323eb434bb6d44e99d709a68d12cca25 => generated 0 bytes in 624 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:39:50.846660 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b639d88b-288b-4dec-9e92-343105ad53cc admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:50.848142 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-b639d88b-288b-4dec-9e92-343105ad53cc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:50.869285 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b639d88b-288b-4dec-9e92-343105ad53cc admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:50.985295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:50.988434 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d21f75f-d555-4052-9adf-fd36c9b8a6d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:50.989465 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d21f75f-d555-4052-9adf-fd36c9b8a6d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:50.989825 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d21f75f-d555-4052-9adf-fd36c9b8a6d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:50.990245 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d21f75f-d555-4052-9adf-fd36c9b8a6d5 admin admin] PATH_INFO: `/v3/projects/fcf615b5ef1e4cf9bdfb7227f38f431d` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:50.992024 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d21f75f-d555-4052-9adf-fd36c9b8a6d5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fcf615b5ef1e4cf9bdfb7227f38f431d)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:50.994549 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d21f75f-d555-4052-9adf-fd36c9b8a6d5 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:51.153736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60cce2a-cc3d-4cd3-a1b2-86bc482e2d03 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:51.157444 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-a60cce2a-cc3d-4cd3-a1b2-86bc482e2d03 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:51.229401 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-6d21f75f-d555-4052-9adf-fd36c9b8a6d5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fcf615b5ef1e4cf9bdfb7227f38f431d', 'request_id': 'req-6d21f75f-d555-4052-9adf-fd36c9b8a6d5'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:51.240037 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-6d21f75f-d555-4052-9adf-fd36c9b8a6d5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fcf615b5ef1e4cf9bdfb7227f38f431d', 'request_id': 'req-6d21f75f-d555-4052-9adf-fd36c9b8a6d5'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:51.285656 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60cce2a-cc3d-4cd3-a1b2-86bc482e2d03 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:51.330815 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-6d21f75f-d555-4052-9adf-fd36c9b8a6d5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fcf615b5ef1e4cf9bdfb7227f38f431d', 'request_id': 'req-6d21f75f-d555-4052-9adf-fd36c9b8a6d5'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:51.345238 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-6d21f75f-d555-4052-9adf-fd36c9b8a6d5 admin admin] The token cache is being invalidate because project fcf615b5ef1e4cf9bdfb7227f38f431d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:39:51.345541 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-6d21f75f-d555-4052-9adf-fd36c9b8a6d5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:51.346344 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-6d21f75f-d555-4052-9adf-fd36c9b8a6d5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:51.348510 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 956/1878] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:39:50 2022] DELETE /identity/v3/projects/fcf615b5ef1e4cf9bdfb7227f38f431d => generated 0 bytes in 502 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:51.349839 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e93cc5b-d266-46aa-ab1d-b81a5d7a645b None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:51.350127 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e93cc5b-d266-46aa-ab1d-b81a5d7a645b None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:51.350640 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e93cc5b-d266-46aa-ab1d-b81a5d7a645b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:51.361360 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47e2787b-8991-40bd-b471-b810b601c781 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:51.362079 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e2787b-8991-40bd-b471-b810b601c781 service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:51.362374 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e2787b-8991-40bd-b471-b810b601c781 service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:51.362711 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47e2787b-8991-40bd-b471-b810b601c781 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:51.365238 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-47e2787b-8991-40bd-b471-b810b601c781 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:51.384879 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-3e93cc5b-d266-46aa-ab1d-b81a5d7a645b None None] MFA Rules not processed for user `2c4deca7942842759f795ab5299b44a3`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:51.386187 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-3e93cc5b-d266-46aa-ab1d-b81a5d7a645b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:51.461385 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e2787b-8991-40bd-b471-b810b601c781 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:51.463328 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47e2787b-8991-40bd-b471-b810b601c781 service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:51.475695 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 923/1879] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:51 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:39:51.477521 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977c84bd-245f-4861-ac5c-9fd58e2a3f49 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:51.478049 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977c84bd-245f-4861-ac5c-9fd58e2a3f49 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:51.478673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977c84bd-245f-4861-ac5c-9fd58e2a3f49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:51.501809 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-977c84bd-245f-4861-ac5c-9fd58e2a3f49 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:51.503519 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-977c84bd-245f-4861-ac5c-9fd58e2a3f49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:51.546985 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 957/1880] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:51 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:51.620059 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 924/1881] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:51.622048 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039e8a37-e01a-4e37-9214-e4df67917117 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:51.622424 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039e8a37-e01a-4e37-9214-e4df67917117 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:51.622841 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039e8a37-e01a-4e37-9214-e4df67917117 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:51.661722 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-039e8a37-e01a-4e37-9214-e4df67917117 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:51.663266 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-039e8a37-e01a-4e37-9214-e4df67917117 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:51.675935 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 958/1882] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:51.683472 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-977c84bd-245f-4861-ac5c-9fd58e2a3f49 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:51.704298 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-977c84bd-245f-4861-ac5c-9fd58e2a3f49 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:51.710309 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6466a012-5db0-4da4-8948-ad6195a60bce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:51.713820 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6466a012-5db0-4da4-8948-ad6195a60bce admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:51.713820 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6466a012-5db0-4da4-8948-ad6195a60bce admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:51.720695 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6466a012-5db0-4da4-8948-ad6195a60bce admin admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:51.723836 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6466a012-5db0-4da4-8948-ad6195a60bce admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:39:51.725798 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6466a012-5db0-4da4-8948-ad6195a60bce admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:51.727793 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6466a012-5db0-4da4-8948-ad6195a60bce admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:51.751289 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 925/1883] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:39:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:39:51.755483 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039e8a37-e01a-4e37-9214-e4df67917117 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:51.768216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039e8a37-e01a-4e37-9214-e4df67917117 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:51.774196 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba42d63a-bc7b-4a7f-955a-35b6d68c2e7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:51.774923 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba42d63a-bc7b-4a7f-955a-35b6d68c2e7d admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:51.775213 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba42d63a-bc7b-4a7f-955a-35b6d68c2e7d admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:51.775506 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba42d63a-bc7b-4a7f-955a-35b6d68c2e7d admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:51.776226 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba42d63a-bc7b-4a7f-955a-35b6d68c2e7d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:51.778673 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba42d63a-bc7b-4a7f-955a-35b6d68c2e7d admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:51.808465 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:51.910949 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 959/1884] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:39:51 2022] POST /identity/v3/projects => generated 387 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:39:51.912329 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6466a012-5db0-4da4-8948-ad6195a60bce admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:51.924573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6466a012-5db0-4da4-8948-ad6195a60bce admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:52.030661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8927c4d2-fa31-437e-92e0-c42b04a3bd01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:52.031777 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8927c4d2-fa31-437e-92e0-c42b04a3bd01 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:52.031777 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8927c4d2-fa31-437e-92e0-c42b04a3bd01 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:52.032238 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8927c4d2-fa31-437e-92e0-c42b04a3bd01 admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:52.032687 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8927c4d2-fa31-437e-92e0-c42b04a3bd01 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:52.034869 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8927c4d2-fa31-437e-92e0-c42b04a3bd01 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:52.115041 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 926/1885] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:39:51 2022] POST /identity/v3/users => generated 294 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:52.119513 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba42d63a-bc7b-4a7f-955a-35b6d68c2e7d admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:52.130707 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba42d63a-bc7b-4a7f-955a-35b6d68c2e7d admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:52.136226 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac148ee-3f50-4ed5-afa0-15580149fdf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:52.137819 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac148ee-3f50-4ed5-afa0-15580149fdf9 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:52.137819 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac148ee-3f50-4ed5-afa0-15580149fdf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:52.137819 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac148ee-3f50-4ed5-afa0-15580149fdf9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:52.138933 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac148ee-3f50-4ed5-afa0-15580149fdf9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:52.140553 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac148ee-3f50-4ed5-afa0-15580149fdf9 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:52.152087 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 960/1886] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:52 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:52.156912 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8927c4d2-fa31-437e-92e0-c42b04a3bd01 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:52.171413 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8927c4d2-fa31-437e-92e0-c42b04a3bd01 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:52.172683 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66bd3bec-e044-4f95-a1a1-45165e42eba7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:52.173431 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bd3bec-e044-4f95-a1a1-45165e42eba7 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:52.173725 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bd3bec-e044-4f95-a1a1-45165e42eba7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:52.174018 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bd3bec-e044-4f95-a1a1-45165e42eba7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:52.174802 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66bd3bec-e044-4f95-a1a1-45165e42eba7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:52.177200 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66bd3bec-e044-4f95-a1a1-45165e42eba7 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:52.187233 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 927/1887] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:52 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:52.191726 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac148ee-3f50-4ed5-afa0-15580149fdf9 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:52.202751 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac148ee-3f50-4ed5-afa0-15580149fdf9 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:52.209917 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c13daa-0cd4-4b95-8c76-8c96ac8be414 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:52.210851 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c13daa-0cd4-4b95-8c76-8c96ac8be414 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:52.211311 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c13daa-0cd4-4b95-8c76-8c96ac8be414 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:52.211595 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c13daa-0cd4-4b95-8c76-8c96ac8be414 admin admin] PATH_INFO: `/v3/projects/20702f06abb8423b8dd150c313be0bde/users/7789bfc9c4de407eb3f00c949d1455da/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:52.228482 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2c13daa-0cd4-4b95-8c76-8c96ac8be414 admin admin] RBAC: Authorizing `identity:create_grant(project_id=20702f06abb8423b8dd150c313be0bde, user_id=7789bfc9c4de407eb3f00c949d1455da, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:52.231121 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2c13daa-0cd4-4b95-8c76-8c96ac8be414 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:52.366732 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 961/1888] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:39:52 2022] PUT /identity/v3/projects/20702f06abb8423b8dd150c313be0bde/users/7789bfc9c4de407eb3f00c949d1455da/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:52.370076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d233a8-a558-4ad3-bd2f-9bc7b7335b77 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:52.370380 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d233a8-a558-4ad3-bd2f-9bc7b7335b77 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:52.370698 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d233a8-a558-4ad3-bd2f-9bc7b7335b77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:52.415691 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-34d233a8-a558-4ad3-bd2f-9bc7b7335b77 None None] MFA Rules not processed for user `7789bfc9c4de407eb3f00c949d1455da`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:52.417711 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-34d233a8-a558-4ad3-bd2f-9bc7b7335b77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:52.514597 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c13daa-0cd4-4b95-8c76-8c96ac8be414 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:52.517941 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-f2c13daa-0cd4-4b95-8c76-8c96ac8be414 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:52.526371 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 928/1889] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:52 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:52.531579 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d233a8-a558-4ad3-bd2f-9bc7b7335b77 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:52.661679 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c13daa-0cd4-4b95-8c76-8c96ac8be414 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:52.673926 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3712b188-c707-4ffc-ab8e-40c0c71bca03 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:52.674709 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3712b188-c707-4ffc-ab8e-40c0c71bca03 service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:52.674991 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3712b188-c707-4ffc-ab8e-40c0c71bca03 service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:52.675339 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3712b188-c707-4ffc-ab8e-40c0c71bca03 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:52.675869 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d233a8-a558-4ad3-bd2f-9bc7b7335b77 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:52.698480 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b725f27b-7fb7-41e0-bc67-9de9011922d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:52.698480 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b725f27b-7fb7-41e0-bc67-9de9011922d2 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:52.698480 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b725f27b-7fb7-41e0-bc67-9de9011922d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:52.698480 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b725f27b-7fb7-41e0-bc67-9de9011922d2 admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:52.699172 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b725f27b-7fb7-41e0-bc67-9de9011922d2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:52.699172 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b725f27b-7fb7-41e0-bc67-9de9011922d2 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:52.793148 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3712b188-c707-4ffc-ab8e-40c0c71bca03 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:52.795230 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3712b188-c707-4ffc-ab8e-40c0c71bca03 service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:52.810759 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 962/1890] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:52 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:39:52.842917 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 929/1891] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:39:52 2022] POST /identity/v3/projects => generated 389 bytes in 313 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:52.847905 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3712b188-c707-4ffc-ab8e-40c0c71bca03 service cinder] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:52.860279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3712b188-c707-4ffc-ab8e-40c0c71bca03 service cinder] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:52.980287 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3452e93-9c1e-4ae4-a311-2e8b38b2f80c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:52.980509 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3452e93-9c1e-4ae4-a311-2e8b38b2f80c admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:52.980880 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3452e93-9c1e-4ae4-a311-2e8b38b2f80c admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:52.981227 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3452e93-9c1e-4ae4-a311-2e8b38b2f80c admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:52.982285 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3452e93-9c1e-4ae4-a311-2e8b38b2f80c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:52.984153 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3452e93-9c1e-4ae4-a311-2e8b38b2f80c admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:53.085312 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=81,keystone:DELETE=18,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:53.109241 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde03e3a-90c7-4eff-8d40-9fd51d8ae253 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:53.110010 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde03e3a-90c7-4eff-8d40-9fd51d8ae253 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:53.110010 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde03e3a-90c7-4eff-8d40-9fd51d8ae253 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:53.111668 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 930/1892] 10.222.0.16 () {54 vars in 921 bytes} [Tue Jul 12 17:39:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:39:53.124682 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961656a6-3f07-4ceb-b39b-8dd0ba2d796d None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:53.124974 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961656a6-3f07-4ceb-b39b-8dd0ba2d796d None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:53.125679 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961656a6-3f07-4ceb-b39b-8dd0ba2d796d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:53.145232 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-961656a6-3f07-4ceb-b39b-8dd0ba2d796d None None] MFA Rules not processed for user `9012d5b88f554dc596582a7693c2a72a`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:53.146560 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-961656a6-3f07-4ceb-b39b-8dd0ba2d796d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:53.155516 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 963/1893] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:39:52 2022] POST /identity/v3/users => generated 301 bytes in 308 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:39:53.159492 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3452e93-9c1e-4ae4-a311-2e8b38b2f80c admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:53.183551 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3452e93-9c1e-4ae4-a311-2e8b38b2f80c admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:53.201744 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea5a7a5-c0bc-475f-a42a-505720768f74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:53.202576 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea5a7a5-c0bc-475f-a42a-505720768f74 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:53.202930 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea5a7a5-c0bc-475f-a42a-505720768f74 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:53.203566 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea5a7a5-c0bc-475f-a42a-505720768f74 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:53.204577 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea5a7a5-c0bc-475f-a42a-505720768f74 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:53.206584 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ea5a7a5-c0bc-475f-a42a-505720768f74 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:53.231256 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 964/1894] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:53 2022] GET /identity/v3/roles => generated 1420 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 12 17:39:53.236343 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea5a7a5-c0bc-475f-a42a-505720768f74 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:53.256565 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ea5a7a5-c0bc-475f-a42a-505720768f74 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:53.264808 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246aa453-8aea-470b-9da5-02f52e0306ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:53.266542 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246aa453-8aea-470b-9da5-02f52e0306ec admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:53.271483 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246aa453-8aea-470b-9da5-02f52e0306ec admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:53.271483 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246aa453-8aea-470b-9da5-02f52e0306ec admin admin] PATH_INFO: `/v3/projects/da6be062768c4af598b14730f2b9b3cc/users/ad990f948ba74184a885ecda992797d5/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:53.290198 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 931/1895] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Jul 12 17:39:53 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:39:53.310156 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-246aa453-8aea-470b-9da5-02f52e0306ec admin admin] RBAC: Authorizing `identity:create_grant(project_id=da6be062768c4af598b14730f2b9b3cc, user_id=ad990f948ba74184a885ecda992797d5, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:53.314100 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-246aa453-8aea-470b-9da5-02f52e0306ec admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:53.332678 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961656a6-3f07-4ceb-b39b-8dd0ba2d796d None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:53.335303 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-961656a6-3f07-4ceb-b39b-8dd0ba2d796d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:53.365096 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961656a6-3f07-4ceb-b39b-8dd0ba2d796d None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:53.371346 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc76d50-6934-4839-beba-37e168b0e653 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:53.372019 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc76d50-6934-4839-beba-37e168b0e653 service nova] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:53.372362 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc76d50-6934-4839-beba-37e168b0e653 service nova] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:53.372679 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc76d50-6934-4839-beba-37e168b0e653 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:53.382442 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc76d50-6934-4839-beba-37e168b0e653 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:53.384136 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc76d50-6934-4839-beba-37e168b0e653 service nova] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:53.396630 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 932/1896] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:53 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:39:53.572099 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 965/1897] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:39:53 2022] PUT /identity/v3/projects/da6be062768c4af598b14730f2b9b3cc/users/ad990f948ba74184a885ecda992797d5/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 338 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:53.577449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc76d50-6934-4839-beba-37e168b0e653 service nova] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:53.593698 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc76d50-6934-4839-beba-37e168b0e653 service nova] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:53.676483 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ab0734-87cf-4afb-ab3e-effee84c98d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:53.676483 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ab0734-87cf-4afb-ab3e-effee84c98d7 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:53.676483 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ab0734-87cf-4afb-ab3e-effee84c98d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:53.676483 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ab0734-87cf-4afb-ab3e-effee84c98d7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:53.676903 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ab0734-87cf-4afb-ab3e-effee84c98d7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:53.677074 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ab0734-87cf-4afb-ab3e-effee84c98d7 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:53.689036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 933/1898] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:39:53 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:39:53.693266 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246aa453-8aea-470b-9da5-02f52e0306ec admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:53.711143 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246aa453-8aea-470b-9da5-02f52e0306ec admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:53.713104 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a015e5-7909-40e5-899f-eb316dcfb723 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:53.714053 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a015e5-7909-40e5-899f-eb316dcfb723 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:53.715124 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a015e5-7909-40e5-899f-eb316dcfb723 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:53.715433 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a015e5-7909-40e5-899f-eb316dcfb723 admin admin] PATH_INFO: `/v3/projects/da6be062768c4af598b14730f2b9b3cc/users/ad990f948ba74184a885ecda992797d5/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:53.718240 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1a015e5-7909-40e5-899f-eb316dcfb723 admin admin] RBAC: Authorizing `identity:create_grant(project_id=da6be062768c4af598b14730f2b9b3cc, user_id=ad990f948ba74184a885ecda992797d5, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:53.720998 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1a015e5-7909-40e5-899f-eb316dcfb723 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:53.744322 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 966/1899] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:39:53 2022] PUT /identity/v3/projects/da6be062768c4af598b14730f2b9b3cc/users/ad990f948ba74184a885ecda992797d5/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:53.756508 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b96401-9733-4d01-83c4-2b27c498d89e None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:53.756709 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b96401-9733-4d01-83c4-2b27c498d89e None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:53.757007 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b96401-9733-4d01-83c4-2b27c498d89e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:53.890337 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-08b96401-9733-4d01-83c4-2b27c498d89e None None] MFA Rules not processed for user `ad990f948ba74184a885ecda992797d5`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:53.892500 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-08b96401-9733-4d01-83c4-2b27c498d89e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:54.024780 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a015e5-7909-40e5-899f-eb316dcfb723 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:54.041615 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 934/1900] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:53 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 288 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:39:54.054731 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b96401-9733-4d01-83c4-2b27c498d89e None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:54.056262 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-08b96401-9733-4d01-83c4-2b27c498d89e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:54.097133 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a015e5-7909-40e5-899f-eb316dcfb723 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:54.101235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7185d1-f1f8-4f1f-996e-fd3798cf4e5d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:54.101905 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7185d1-f1f8-4f1f-996e-fd3798cf4e5d service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:54.103046 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7185d1-f1f8-4f1f-996e-fd3798cf4e5d service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:54.103046 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7185d1-f1f8-4f1f-996e-fd3798cf4e5d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:54.104890 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-ca7185d1-f1f8-4f1f-996e-fd3798cf4e5d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:54.134035 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b96401-9733-4d01-83c4-2b27c498d89e None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:54.139115 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e391d02e-43f0-4a34-bc4f-230de12fad16 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:54.140103 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e391d02e-43f0-4a34-bc4f-230de12fad16 service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:54.140365 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e391d02e-43f0-4a34-bc4f-230de12fad16 service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:54.140645 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e391d02e-43f0-4a34-bc4f-230de12fad16 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:54.152735 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e391d02e-43f0-4a34-bc4f-230de12fad16 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:54.154685 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e391d02e-43f0-4a34-bc4f-230de12fad16 service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:54.165907 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 935/1901] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:54 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:39:54.195856 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7185d1-f1f8-4f1f-996e-fd3798cf4e5d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:54.197500 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7185d1-f1f8-4f1f-996e-fd3798cf4e5d service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:54.221413 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 967/1902] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:54 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 196 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:39:54.729393 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e391d02e-43f0-4a34-bc4f-230de12fad16 service cinder] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:54.850035 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e391d02e-43f0-4a34-bc4f-230de12fad16 service cinder] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:54.859475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3587821a-05a2-4466-bc6c-cf9628ac09ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:54.860656 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3587821a-05a2-4466-bc6c-cf9628ac09ad admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:54.861144 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3587821a-05a2-4466-bc6c-cf9628ac09ad admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:54.861706 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3587821a-05a2-4466-bc6c-cf9628ac09ad admin admin] PATH_INFO: `/v3/users/7789bfc9c4de407eb3f00c949d1455da` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:54.871378 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3587821a-05a2-4466-bc6c-cf9628ac09ad admin admin] RBAC: Authorizing `identity:delete_user(user_id=7789bfc9c4de407eb3f00c949d1455da)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:54.874888 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3587821a-05a2-4466-bc6c-cf9628ac09ad admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:54.880594 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:55.101338 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77413c7d-6b39-488b-82c5-aaa5875164ce None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:55.101921 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77413c7d-6b39-488b-82c5-aaa5875164ce None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:55.102018 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77413c7d-6b39-488b-82c5-aaa5875164ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:55.129338 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-77413c7d-6b39-488b-82c5-aaa5875164ce None None] MFA Rules not processed for user `2c4deca7942842759f795ab5299b44a3`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:55.130724 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-77413c7d-6b39-488b-82c5-aaa5875164ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:55.140589 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-3587821a-05a2-4466-bc6c-cf9628ac09ad admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7789bfc9c4de407eb3f00c949d1455da'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:55.202790 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 968/1903] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:55 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:39:55.274516 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-3587821a-05a2-4466-bc6c-cf9628ac09ad admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7789bfc9c4de407eb3f00c949d1455da'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:55.275916 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-3587821a-05a2-4466-bc6c-cf9628ac09ad admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7789bfc9c4de407eb3f00c949d1455da'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:55.287514 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-3587821a-05a2-4466-bc6c-cf9628ac09ad admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7789bfc9c4de407eb3f00c949d1455da'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:55.314578 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-3587821a-05a2-4466-bc6c-cf9628ac09ad admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7789bfc9c4de407eb3f00c949d1455da'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:55.323503 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 936/1904] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:39:54 2022] DELETE /identity/v3/users/7789bfc9c4de407eb3f00c949d1455da => generated 0 bytes in 595 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:55.330793 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43501224-b4e3-4775-ab21-37a76545d39f None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:55.331627 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43501224-b4e3-4775-ab21-37a76545d39f None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:55.332537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43501224-b4e3-4775-ab21-37a76545d39f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:55.356167 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-43501224-b4e3-4775-ab21-37a76545d39f None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:55.357588 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-43501224-b4e3-4775-ab21-37a76545d39f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:55.447629 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 969/1905] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:55.464094 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3587821a-05a2-4466-bc6c-cf9628ac09ad admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:55.465668 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-3587821a-05a2-4466-bc6c-cf9628ac09ad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:55.589973 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3587821a-05a2-4466-bc6c-cf9628ac09ad admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:55.600907 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb167c6a-1e05-46a3-8e47-90d933d096b0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:55.601936 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb167c6a-1e05-46a3-8e47-90d933d096b0 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:55.602444 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb167c6a-1e05-46a3-8e47-90d933d096b0 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:55.603124 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb167c6a-1e05-46a3-8e47-90d933d096b0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:55.623623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb167c6a-1e05-46a3-8e47-90d933d096b0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:55.626609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb167c6a-1e05-46a3-8e47-90d933d096b0 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:55.670528 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 937/1906] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:39:55 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 207 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:39:56.456999 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=96,keystone:DELETE=2,keystone:INSERT=2 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:56.762219 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43501224-b4e3-4775-ab21-37a76545d39f None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:56.766954 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-43501224-b4e3-4775-ab21-37a76545d39f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:56.835250 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43501224-b4e3-4775-ab21-37a76545d39f None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:56.860074 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea254d1d-0d7e-4fdb-bc69-a7a4f0d6919c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:56.864263 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea254d1d-0d7e-4fdb-bc69-a7a4f0d6919c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:56.865412 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea254d1d-0d7e-4fdb-bc69-a7a4f0d6919c admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:56.866982 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea254d1d-0d7e-4fdb-bc69-a7a4f0d6919c admin admin] PATH_INFO: `/v3/projects/20702f06abb8423b8dd150c313be0bde` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:56.869752 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea254d1d-0d7e-4fdb-bc69-a7a4f0d6919c admin admin] RBAC: Authorizing `identity:delete_project(project_id=20702f06abb8423b8dd150c313be0bde)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:56.875085 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea254d1d-0d7e-4fdb-bc69-a7a4f0d6919c admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:57.485217 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:39:57.504938 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ea254d1d-0d7e-4fdb-bc69-a7a4f0d6919c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '20702f06abb8423b8dd150c313be0bde', 'request_id': 'req-ea254d1d-0d7e-4fdb-bc69-a7a4f0d6919c'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:57.511692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ea254d1d-0d7e-4fdb-bc69-a7a4f0d6919c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '20702f06abb8423b8dd150c313be0bde', 'request_id': 'req-ea254d1d-0d7e-4fdb-bc69-a7a4f0d6919c'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:57.643374 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ea254d1d-0d7e-4fdb-bc69-a7a4f0d6919c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '20702f06abb8423b8dd150c313be0bde', 'request_id': 'req-ea254d1d-0d7e-4fdb-bc69-a7a4f0d6919c'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:57.696358 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ea254d1d-0d7e-4fdb-bc69-a7a4f0d6919c admin admin] The token cache is being invalidate because project 20702f06abb8423b8dd150c313be0bde was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:39:57.697300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ea254d1d-0d7e-4fdb-bc69-a7a4f0d6919c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:57.698500 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ea254d1d-0d7e-4fdb-bc69-a7a4f0d6919c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:57.704462 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 970/1907] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:39:56 2022] DELETE /identity/v3/projects/20702f06abb8423b8dd150c313be0bde => generated 0 bytes in 941 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:39:57.705959 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb167c6a-1e05-46a3-8e47-90d933d096b0 service neutron] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:57.717428 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb167c6a-1e05-46a3-8e47-90d933d096b0 service neutron] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:57.796313 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f603d2-4f8a-41a6-a299-84bafbf69a91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:57.798015 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f603d2-4f8a-41a6-a299-84bafbf69a91 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:57.798612 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f603d2-4f8a-41a6-a299-84bafbf69a91 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:57.799136 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17f603d2-4f8a-41a6-a299-84bafbf69a91 admin admin] PATH_INFO: `/v3/users/ad990f948ba74184a885ecda992797d5` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:57.801486 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f603d2-4f8a-41a6-a299-84bafbf69a91 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ad990f948ba74184a885ecda992797d5)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:57.804918 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17f603d2-4f8a-41a6-a299-84bafbf69a91 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:58.083500 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-17f603d2-4f8a-41a6-a299-84bafbf69a91 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ad990f948ba74184a885ecda992797d5'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:58.180305 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-17f603d2-4f8a-41a6-a299-84bafbf69a91 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ad990f948ba74184a885ecda992797d5'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:58.181019 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-17f603d2-4f8a-41a6-a299-84bafbf69a91 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ad990f948ba74184a885ecda992797d5'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:58.193843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-17f603d2-4f8a-41a6-a299-84bafbf69a91 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ad990f948ba74184a885ecda992797d5'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:58.218316 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-17f603d2-4f8a-41a6-a299-84bafbf69a91 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ad990f948ba74184a885ecda992797d5'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:58.228063 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 938/1908] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:39:57 2022] DELETE /identity/v3/users/ad990f948ba74184a885ecda992797d5 => generated 0 bytes in 523 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:59.100275 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea254d1d-0d7e-4fdb-bc69-a7a4f0d6919c admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:59.102591 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-ea254d1d-0d7e-4fdb-bc69-a7a4f0d6919c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:59.129908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea254d1d-0d7e-4fdb-bc69-a7a4f0d6919c admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:59.236689 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf2f8a9-daaf-4ac0-b283-af2bf421440e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:59.237855 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf2f8a9-daaf-4ac0-b283-af2bf421440e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:59.238321 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf2f8a9-daaf-4ac0-b283-af2bf421440e admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:59.238860 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf2f8a9-daaf-4ac0-b283-af2bf421440e admin admin] PATH_INFO: `/v3/projects/da6be062768c4af598b14730f2b9b3cc` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:59.241375 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf2f8a9-daaf-4ac0-b283-af2bf421440e admin admin] RBAC: Authorizing `identity:delete_project(project_id=da6be062768c4af598b14730f2b9b3cc)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:59.243655 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf2f8a9-daaf-4ac0-b283-af2bf421440e admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:59.440495 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-fbf2f8a9-daaf-4ac0-b283-af2bf421440e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'da6be062768c4af598b14730f2b9b3cc', 'request_id': 'req-fbf2f8a9-daaf-4ac0-b283-af2bf421440e'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:59.447702 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-fbf2f8a9-daaf-4ac0-b283-af2bf421440e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'da6be062768c4af598b14730f2b9b3cc', 'request_id': 'req-fbf2f8a9-daaf-4ac0-b283-af2bf421440e'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:59.668884 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f603d2-4f8a-41a6-a299-84bafbf69a91 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:59.670728 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-17f603d2-4f8a-41a6-a299-84bafbf69a91 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:59.682045 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-fbf2f8a9-daaf-4ac0-b283-af2bf421440e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'da6be062768c4af598b14730f2b9b3cc', 'request_id': 'req-fbf2f8a9-daaf-4ac0-b283-af2bf421440e'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:59.695575 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17f603d2-4f8a-41a6-a299-84bafbf69a91 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:59.701592 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-fbf2f8a9-daaf-4ac0-b283-af2bf421440e admin admin] The token cache is being invalidate because project da6be062768c4af598b14730f2b9b3cc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:39:59.702689 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-fbf2f8a9-daaf-4ac0-b283-af2bf421440e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:59.704827 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-fbf2f8a9-daaf-4ac0-b283-af2bf421440e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:39:59.708252 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 971/1909] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:39:59 2022] DELETE /identity/v3/projects/da6be062768c4af598b14730f2b9b3cc => generated 0 bytes in 609 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:39:59.714626 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db82dab9-f484-486c-9c2b-401d7127366d None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:59.714875 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db82dab9-f484-486c-9c2b-401d7127366d None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:59.715183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db82dab9-f484-486c-9c2b-401d7127366d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:59.756612 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-db82dab9-f484-486c-9c2b-401d7127366d None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:59.758517 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-db82dab9-f484-486c-9c2b-401d7127366d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:59.803774 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d6b8b7-deb7-4345-b3c6-1e8e9167dc81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:59.804710 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d6b8b7-deb7-4345-b3c6-1e8e9167dc81 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:59.805303 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d6b8b7-deb7-4345-b3c6-1e8e9167dc81 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:59.805603 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d6b8b7-deb7-4345-b3c6-1e8e9167dc81 admin admin] PATH_INFO: `/v3/users/fe88c6cc9f1a4ef4be4f57d93864ac3f` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:59.807390 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d6b8b7-deb7-4345-b3c6-1e8e9167dc81 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fe88c6cc9f1a4ef4be4f57d93864ac3f)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:59.811491 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d6b8b7-deb7-4345-b3c6-1e8e9167dc81 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:59.882238 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 972/1910] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:59 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:39:59.891202 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62151df2-ab46-4cf0-8d38-48556e7cb2c8 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:59.891783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62151df2-ab46-4cf0-8d38-48556e7cb2c8 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:59.892070 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62151df2-ab46-4cf0-8d38-48556e7cb2c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:59.915138 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-62151df2-ab46-4cf0-8d38-48556e7cb2c8 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:39:59.916388 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-62151df2-ab46-4cf0-8d38-48556e7cb2c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:39:59.927670 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 973/1911] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:39:59 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:39:59.932488 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62151df2-ab46-4cf0-8d38-48556e7cb2c8 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:39:59.948290 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62151df2-ab46-4cf0-8d38-48556e7cb2c8 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:39:59.955753 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199934cb-c013-4c7c-8b1f-a92c63a3229f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:39:59.962084 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199934cb-c013-4c7c-8b1f-a92c63a3229f admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:39:59.962084 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199934cb-c013-4c7c-8b1f-a92c63a3229f admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:39:59.962084 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199934cb-c013-4c7c-8b1f-a92c63a3229f admin admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:39:59.962084 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-199934cb-c013-4c7c-8b1f-a92c63a3229f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:39:59.962084 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-199934cb-c013-4c7c-8b1f-a92c63a3229f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:39:59.962084 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-199934cb-c013-4c7c-8b1f-a92c63a3229f admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:39:59.979527 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 974/1912] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:39:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:39:59.988745 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199934cb-c013-4c7c-8b1f-a92c63a3229f admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:00.011976 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199934cb-c013-4c7c-8b1f-a92c63a3229f admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:00.051705 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab75c95-675d-4766-a53f-29bfc410f509 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:00.053556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab75c95-675d-4766-a53f-29bfc410f509 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:00.060296 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-f4d6b8b7-deb7-4345-b3c6-1e8e9167dc81 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fe88c6cc9f1a4ef4be4f57d93864ac3f'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:00.063525 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab75c95-675d-4766-a53f-29bfc410f509 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:00.064139 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab75c95-675d-4766-a53f-29bfc410f509 admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:00.066207 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ab75c95-675d-4766-a53f-29bfc410f509 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:00.070405 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ab75c95-675d-4766-a53f-29bfc410f509 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:00.181712 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-f4d6b8b7-deb7-4345-b3c6-1e8e9167dc81 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fe88c6cc9f1a4ef4be4f57d93864ac3f'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:00.182779 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-f4d6b8b7-deb7-4345-b3c6-1e8e9167dc81 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fe88c6cc9f1a4ef4be4f57d93864ac3f'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:00.195665 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-f4d6b8b7-deb7-4345-b3c6-1e8e9167dc81 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fe88c6cc9f1a4ef4be4f57d93864ac3f'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:00.217032 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-f4d6b8b7-deb7-4345-b3c6-1e8e9167dc81 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fe88c6cc9f1a4ef4be4f57d93864ac3f'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:00.229667 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 939/1913] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:39:59 2022] DELETE /identity/v3/users/fe88c6cc9f1a4ef4be4f57d93864ac3f => generated 0 bytes in 561 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:40:00.244979 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 975/1914] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:39:59 2022] POST /identity/v3/projects => generated 359 bytes in 261 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:40:00.250565 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a3c563-d584-4d5d-92ca-b9e659d24a15 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:00.251306 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a3c563-d584-4d5d-92ca-b9e659d24a15 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:00.251495 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab75c95-675d-4766-a53f-29bfc410f509 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:00.252008 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a3c563-d584-4d5d-92ca-b9e659d24a15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:00.255442 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-1ab75c95-675d-4766-a53f-29bfc410f509 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:00.276319 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-f7a3c563-d584-4d5d-92ca-b9e659d24a15 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:40:00.279218 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-f7a3c563-d584-4d5d-92ca-b9e659d24a15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:00.382604 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 940/1915] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:40:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:40:00.398904 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a3c563-d584-4d5d-92ca-b9e659d24a15 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:00.401127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-f7a3c563-d584-4d5d-92ca-b9e659d24a15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:00.419561 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab75c95-675d-4766-a53f-29bfc410f509 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:00.440863 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d72023e-e5ac-4d04-a0f4-15743f4357f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:00.454642 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d72023e-e5ac-4d04-a0f4-15743f4357f3 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:00.454838 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d72023e-e5ac-4d04-a0f4-15743f4357f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:00.456262 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d72023e-e5ac-4d04-a0f4-15743f4357f3 admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:00.457087 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d72023e-e5ac-4d04-a0f4-15743f4357f3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:00.459924 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d72023e-e5ac-4d04-a0f4-15743f4357f3 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:00.495521 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a3c563-d584-4d5d-92ca-b9e659d24a15 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:00.500647 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ccec33-dc9c-4fcf-92d0-d7cd945f7bc3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:00.501340 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ccec33-dc9c-4fcf-92d0-d7cd945f7bc3 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:00.501625 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ccec33-dc9c-4fcf-92d0-d7cd945f7bc3 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:00.501906 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ccec33-dc9c-4fcf-92d0-d7cd945f7bc3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:00.512714 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15ccec33-dc9c-4fcf-92d0-d7cd945f7bc3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:00.515934 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15ccec33-dc9c-4fcf-92d0-d7cd945f7bc3 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:00.524750 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 941/1916] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:40:00 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:40:00.583061 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:40:00.591146 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 976/1917] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:40:00 2022] POST /identity/v3/users => generated 280 bytes in 340 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:40:00.595543 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ccec33-dc9c-4fcf-92d0-d7cd945f7bc3 service neutron] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:00.616187 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ccec33-dc9c-4fcf-92d0-d7cd945f7bc3 service neutron] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:00.624667 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a8a912-dd72-4e75-b425-c8467356933f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:00.627990 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a8a912-dd72-4e75-b425-c8467356933f admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:00.627990 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a8a912-dd72-4e75-b425-c8467356933f admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:00.627990 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a8a912-dd72-4e75-b425-c8467356933f admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:00.627990 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a8a912-dd72-4e75-b425-c8467356933f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:00.630492 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a8a912-dd72-4e75-b425-c8467356933f admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:00.646503 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 942/1918] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:40:00 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:40:00.651356 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d72023e-e5ac-4d04-a0f4-15743f4357f3 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:00.674604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d72023e-e5ac-4d04-a0f4-15743f4357f3 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:00.679692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-953b8b13-1276-492b-a985-a2281ce327d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:00.680213 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953b8b13-1276-492b-a985-a2281ce327d4 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:00.680502 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953b8b13-1276-492b-a985-a2281ce327d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:00.680789 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953b8b13-1276-492b-a985-a2281ce327d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:00.681545 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-953b8b13-1276-492b-a985-a2281ce327d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:00.683312 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-953b8b13-1276-492b-a985-a2281ce327d4 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:00.702419 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 977/1919] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:40:00 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:40:00.709453 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a8a912-dd72-4e75-b425-c8467356933f admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:00.725852 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a8a912-dd72-4e75-b425-c8467356933f admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:00.732022 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528570a6-fcf3-4e5d-83f4-08aa1d997309 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:00.732907 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528570a6-fcf3-4e5d-83f4-08aa1d997309 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:00.733202 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528570a6-fcf3-4e5d-83f4-08aa1d997309 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:00.733504 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528570a6-fcf3-4e5d-83f4-08aa1d997309 admin admin] PATH_INFO: `/v3/projects/a7f8182c83504c0e9232e34859464470/users/995368ff69a3440fb951371535d10cf5/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:00.751689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-528570a6-fcf3-4e5d-83f4-08aa1d997309 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a7f8182c83504c0e9232e34859464470, user_id=995368ff69a3440fb951371535d10cf5, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:00.754401 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-528570a6-fcf3-4e5d-83f4-08aa1d997309 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:00.880874 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 943/1920] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:40:00 2022] PUT /identity/v3/projects/a7f8182c83504c0e9232e34859464470/users/995368ff69a3440fb951371535d10cf5/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:40:00.894311 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f6e6ef-8364-4d39-ad86-c6e4ad7109e7 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:00.894676 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f6e6ef-8364-4d39-ad86-c6e4ad7109e7 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:00.894984 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f6e6ef-8364-4d39-ad86-c6e4ad7109e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:00.929506 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-49f6e6ef-8364-4d39-ad86-c6e4ad7109e7 None None] MFA Rules not processed for user `995368ff69a3440fb951371535d10cf5`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:40:00.930809 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-49f6e6ef-8364-4d39-ad86-c6e4ad7109e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:01.018391 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 978/1921] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:40:00 2022] POST /identity/v3/auth/tokens => generated 3226 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:40:01.078067 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528570a6-fcf3-4e5d-83f4-08aa1d997309 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:01.080761 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-528570a6-fcf3-4e5d-83f4-08aa1d997309 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:01.145643 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f6e6ef-8364-4d39-ad86-c6e4ad7109e7 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:01.148987 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-49f6e6ef-8364-4d39-ad86-c6e4ad7109e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:01.150465 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:40:01.187592 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528570a6-fcf3-4e5d-83f4-08aa1d997309 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:01.201916 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3eb4c0a-bb68-41ef-a032-b5290b473ced service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:01.202647 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3eb4c0a-bb68-41ef-a032-b5290b473ced service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:01.202936 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3eb4c0a-bb68-41ef-a032-b5290b473ced service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:01.203211 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3eb4c0a-bb68-41ef-a032-b5290b473ced service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:01.215316 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3eb4c0a-bb68-41ef-a032-b5290b473ced service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:01.218105 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3eb4c0a-bb68-41ef-a032-b5290b473ced service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:01.230780 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 944/1922] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:40:01 2022] GET /identity/v3/auth/tokens => generated 3226 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:40:01.255977 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3eb4c0a-bb68-41ef-a032-b5290b473ced service cinder] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:01.322242 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f6e6ef-8364-4d39-ad86-c6e4ad7109e7 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:01.332899 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5951c8d8-43c4-46b3-bfef-5d6851803297 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:01.335730 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5951c8d8-43c4-46b3-bfef-5d6851803297 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:01.336201 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5951c8d8-43c4-46b3-bfef-5d6851803297 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:01.336689 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5951c8d8-43c4-46b3-bfef-5d6851803297 admin admin] PATH_INFO: `/v3/projects/2896afab75b54d8eb3692d56ea153bf5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:01.340043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5951c8d8-43c4-46b3-bfef-5d6851803297 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2896afab75b54d8eb3692d56ea153bf5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:01.345000 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5951c8d8-43c4-46b3-bfef-5d6851803297 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:01.406275 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3eb4c0a-bb68-41ef-a032-b5290b473ced service cinder] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:01.429861 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9799446-d5e9-47ed-93ab-d18dac8f5582 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:01.430840 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9799446-d5e9-47ed-93ab-d18dac8f5582 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:01.431137 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9799446-d5e9-47ed-93ab-d18dac8f5582 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:01.435800 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9799446-d5e9-47ed-93ab-d18dac8f5582 admin admin] PATH_INFO: `/v3/users/995368ff69a3440fb951371535d10cf5` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:01.435800 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9799446-d5e9-47ed-93ab-d18dac8f5582 admin admin] RBAC: Authorizing `identity:delete_user(user_id=995368ff69a3440fb951371535d10cf5)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:01.437257 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9799446-d5e9-47ed-93ab-d18dac8f5582 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:01.534689 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-5951c8d8-43c4-46b3-bfef-5d6851803297 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2896afab75b54d8eb3692d56ea153bf5', 'request_id': 'req-5951c8d8-43c4-46b3-bfef-5d6851803297'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:01.541010 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-5951c8d8-43c4-46b3-bfef-5d6851803297 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2896afab75b54d8eb3692d56ea153bf5', 'request_id': 'req-5951c8d8-43c4-46b3-bfef-5d6851803297'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:01.633696 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-5951c8d8-43c4-46b3-bfef-5d6851803297 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2896afab75b54d8eb3692d56ea153bf5', 'request_id': 'req-5951c8d8-43c4-46b3-bfef-5d6851803297'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:01.648871 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-5951c8d8-43c4-46b3-bfef-5d6851803297 admin admin] The token cache is being invalidate because project 2896afab75b54d8eb3692d56ea153bf5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:40:01.649200 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-5951c8d8-43c4-46b3-bfef-5d6851803297 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:01.649945 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-5951c8d8-43c4-46b3-bfef-5d6851803297 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:01.651753 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 979/1923] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:40:01 2022] DELETE /identity/v3/projects/2896afab75b54d8eb3692d56ea153bf5 => generated 0 bytes in 507 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:40:01.657644 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5951c8d8-43c4-46b3-bfef-5d6851803297 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:01.659297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-5951c8d8-43c4-46b3-bfef-5d6851803297 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:01.729551 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5951c8d8-43c4-46b3-bfef-5d6851803297 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:01.734748 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae5ff23-011b-4390-9241-8c14ac3cc1d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:01.735633 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae5ff23-011b-4390-9241-8c14ac3cc1d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:01.735926 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae5ff23-011b-4390-9241-8c14ac3cc1d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:01.736208 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae5ff23-011b-4390-9241-8c14ac3cc1d5 admin admin] PATH_INFO: `/v3/users/b5531745ed794253858009c00754c5a4` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:01.737456 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae5ff23-011b-4390-9241-8c14ac3cc1d5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b5531745ed794253858009c00754c5a4)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:01.739442 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae5ff23-011b-4390-9241-8c14ac3cc1d5 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:01.820492 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-e9799446-d5e9-47ed-93ab-d18dac8f5582 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '995368ff69a3440fb951371535d10cf5'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:01.964696 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-e9799446-d5e9-47ed-93ab-d18dac8f5582 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '995368ff69a3440fb951371535d10cf5'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:01.965501 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-e9799446-d5e9-47ed-93ab-d18dac8f5582 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '995368ff69a3440fb951371535d10cf5'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:01.986763 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-e9799446-d5e9-47ed-93ab-d18dac8f5582 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '995368ff69a3440fb951371535d10cf5'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:02.033889 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-e9799446-d5e9-47ed-93ab-d18dac8f5582 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '995368ff69a3440fb951371535d10cf5'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:02.046819 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 945/1924] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:40:01 2022] DELETE /identity/v3/users/995368ff69a3440fb951371535d10cf5 => generated 0 bytes in 791 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:40:02.052064 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc7aa1b-6512-4e93-8f00-55a1178da7c3 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:02.057786 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc7aa1b-6512-4e93-8f00-55a1178da7c3 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:02.058133 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc7aa1b-6512-4e93-8f00-55a1178da7c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:02.125957 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-aae5ff23-011b-4390-9241-8c14ac3cc1d5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b5531745ed794253858009c00754c5a4'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:02.127284 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-7fc7aa1b-6512-4e93-8f00-55a1178da7c3 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:40:02.128298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-aae5ff23-011b-4390-9241-8c14ac3cc1d5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b5531745ed794253858009c00754c5a4'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:02.128922 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-7fc7aa1b-6512-4e93-8f00-55a1178da7c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:02.133650 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-aae5ff23-011b-4390-9241-8c14ac3cc1d5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b5531745ed794253858009c00754c5a4'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:02.160566 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-aae5ff23-011b-4390-9241-8c14ac3cc1d5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b5531745ed794253858009c00754c5a4'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:02.196389 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-aae5ff23-011b-4390-9241-8c14ac3cc1d5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b5531745ed794253858009c00754c5a4'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:02.263226 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 946/1925] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:40:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:40:02.297393 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc7aa1b-6512-4e93-8f00-55a1178da7c3 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:02.304399 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-7fc7aa1b-6512-4e93-8f00-55a1178da7c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:02.370641 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 980/1926] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:40:01 2022] DELETE /identity/v3/users/b5531745ed794253858009c00754c5a4 => generated 0 bytes in 714 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:40:02.410172 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc7aa1b-6512-4e93-8f00-55a1178da7c3 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:02.486065 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d28b315-bd7f-4507-809d-fdb69498fd08 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:02.486866 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d28b315-bd7f-4507-809d-fdb69498fd08 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:02.487224 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d28b315-bd7f-4507-809d-fdb69498fd08 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:02.487594 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d28b315-bd7f-4507-809d-fdb69498fd08 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:02.597034 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d28b315-bd7f-4507-809d-fdb69498fd08 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:02.599317 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d28b315-bd7f-4507-809d-fdb69498fd08 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:02.628578 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 947/1927] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:40:02 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:40:03.399542 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:40:03.637593 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae5ff23-011b-4390-9241-8c14ac3cc1d5 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:03.639012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-aae5ff23-011b-4390-9241-8c14ac3cc1d5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:03.655883 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae5ff23-011b-4390-9241-8c14ac3cc1d5 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:03.662167 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31a197e-ce4e-4301-863f-4c7a0fc31c90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:03.663395 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31a197e-ce4e-4301-863f-4c7a0fc31c90 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:03.663829 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31a197e-ce4e-4301-863f-4c7a0fc31c90 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:03.664258 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31a197e-ce4e-4301-863f-4c7a0fc31c90 admin admin] PATH_INFO: `/v3/projects/18f82954e6a348c9b4d5db65b70a119c` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:03.666053 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31a197e-ce4e-4301-863f-4c7a0fc31c90 admin admin] RBAC: Authorizing `identity:delete_project(project_id=18f82954e6a348c9b4d5db65b70a119c)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:03.669537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31a197e-ce4e-4301-863f-4c7a0fc31c90 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:03.680525 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:40:03.882630 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-d31a197e-ce4e-4301-863f-4c7a0fc31c90 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '18f82954e6a348c9b4d5db65b70a119c', 'request_id': 'req-d31a197e-ce4e-4301-863f-4c7a0fc31c90'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:03.890832 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-d31a197e-ce4e-4301-863f-4c7a0fc31c90 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '18f82954e6a348c9b4d5db65b70a119c', 'request_id': 'req-d31a197e-ce4e-4301-863f-4c7a0fc31c90'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:03.931179 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d28b315-bd7f-4507-809d-fdb69498fd08 service neutron] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:03.933774 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-1d28b315-bd7f-4507-809d-fdb69498fd08 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:03.959707 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d28b315-bd7f-4507-809d-fdb69498fd08 service neutron] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:03.995819 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-d31a197e-ce4e-4301-863f-4c7a0fc31c90 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '18f82954e6a348c9b4d5db65b70a119c', 'request_id': 'req-d31a197e-ce4e-4301-863f-4c7a0fc31c90'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:04.024075 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-d31a197e-ce4e-4301-863f-4c7a0fc31c90 admin admin] The token cache is being invalidate because project 18f82954e6a348c9b4d5db65b70a119c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:40:04.024075 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-d31a197e-ce4e-4301-863f-4c7a0fc31c90 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:04.024075 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-d31a197e-ce4e-4301-863f-4c7a0fc31c90 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:04.026016 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 981/1928] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:40:03 2022] DELETE /identity/v3/projects/18f82954e6a348c9b4d5db65b70a119c => generated 0 bytes in 389 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:40:04.030209 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31a197e-ce4e-4301-863f-4c7a0fc31c90 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:04.039309 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-d31a197e-ce4e-4301-863f-4c7a0fc31c90 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:04.135046 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1fe8f42-8c6c-4096-b9f1-5bdfdbae8885 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:04.141207 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fe8f42-8c6c-4096-b9f1-5bdfdbae8885 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:04.143607 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fe8f42-8c6c-4096-b9f1-5bdfdbae8885 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:04.145637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fe8f42-8c6c-4096-b9f1-5bdfdbae8885 admin admin] PATH_INFO: `/v3/projects/a7f8182c83504c0e9232e34859464470` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:04.155073 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1fe8f42-8c6c-4096-b9f1-5bdfdbae8885 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a7f8182c83504c0e9232e34859464470)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:04.166217 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1fe8f42-8c6c-4096-b9f1-5bdfdbae8885 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:04.191221 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31a197e-ce4e-4301-863f-4c7a0fc31c90 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:04.206523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e05e419-fcf3-416c-b32b-682687b6cda6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:04.207644 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e05e419-fcf3-416c-b32b-682687b6cda6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:04.208146 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e05e419-fcf3-416c-b32b-682687b6cda6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:04.208579 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e05e419-fcf3-416c-b32b-682687b6cda6 admin admin] PATH_INFO: `/v3/users/e9eaf24f35a14c3489341eb0ccd0a53d` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:04.210477 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e05e419-fcf3-416c-b32b-682687b6cda6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e9eaf24f35a14c3489341eb0ccd0a53d)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:04.213867 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e05e419-fcf3-416c-b32b-682687b6cda6 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:04.314365 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-a1fe8f42-8c6c-4096-b9f1-5bdfdbae8885 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a7f8182c83504c0e9232e34859464470', 'request_id': 'req-a1fe8f42-8c6c-4096-b9f1-5bdfdbae8885'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:04.321756 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-a1fe8f42-8c6c-4096-b9f1-5bdfdbae8885 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a7f8182c83504c0e9232e34859464470', 'request_id': 'req-a1fe8f42-8c6c-4096-b9f1-5bdfdbae8885'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:04.328598 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-a1fe8f42-8c6c-4096-b9f1-5bdfdbae8885 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a7f8182c83504c0e9232e34859464470', 'request_id': 'req-a1fe8f42-8c6c-4096-b9f1-5bdfdbae8885'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:04.462702 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-a1fe8f42-8c6c-4096-b9f1-5bdfdbae8885 admin admin] The token cache is being invalidate because project a7f8182c83504c0e9232e34859464470 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:40:04.467846 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-a1fe8f42-8c6c-4096-b9f1-5bdfdbae8885 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:04.467846 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-a1fe8f42-8c6c-4096-b9f1-5bdfdbae8885 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:04.467846 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 948/1929] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:40:03 2022] DELETE /identity/v3/projects/a7f8182c83504c0e9232e34859464470 => generated 0 bytes in 536 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:40:04.477399 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f3067d-10d2-42a7-9a7e-65de203447a4 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:04.477715 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f3067d-10d2-42a7-9a7e-65de203447a4 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:04.478314 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f3067d-10d2-42a7-9a7e-65de203447a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:04.522309 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-35f3067d-10d2-42a7-9a7e-65de203447a4 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:40:04.523272 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-0e05e419-fcf3-416c-b32b-682687b6cda6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e9eaf24f35a14c3489341eb0ccd0a53d'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:04.524245 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-0e05e419-fcf3-416c-b32b-682687b6cda6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e9eaf24f35a14c3489341eb0ccd0a53d'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:04.525129 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-35f3067d-10d2-42a7-9a7e-65de203447a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:04.525249 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-0e05e419-fcf3-416c-b32b-682687b6cda6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e9eaf24f35a14c3489341eb0ccd0a53d'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:04.534994 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-0e05e419-fcf3-416c-b32b-682687b6cda6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e9eaf24f35a14c3489341eb0ccd0a53d'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:04.551482 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-0e05e419-fcf3-416c-b32b-682687b6cda6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e9eaf24f35a14c3489341eb0ccd0a53d'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:04.634577 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 949/1930] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:40:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:40:04.643829 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a879088b-7d4d-4e7d-a898-3db12feb26f7 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:04.644491 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a879088b-7d4d-4e7d-a898-3db12feb26f7 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:04.644999 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a879088b-7d4d-4e7d-a898-3db12feb26f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:04.646740 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 982/1931] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:40:04 2022] DELETE /identity/v3/users/e9eaf24f35a14c3489341eb0ccd0a53d => generated 0 bytes in 617 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:40:04.688205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-a879088b-7d4d-4e7d-a898-3db12feb26f7 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:40:04.689680 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-a879088b-7d4d-4e7d-a898-3db12feb26f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:04.796402 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 950/1932] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:40:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:40:04.800910 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e05e419-fcf3-416c-b32b-682687b6cda6 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:04.812861 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e05e419-fcf3-416c-b32b-682687b6cda6 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:04.818231 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c775a8-acfa-4bad-a4bf-a61143f7bff3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:04.819458 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c775a8-acfa-4bad-a4bf-a61143f7bff3 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:04.819930 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c775a8-acfa-4bad-a4bf-a61143f7bff3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:04.820653 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c775a8-acfa-4bad-a4bf-a61143f7bff3 admin admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:04.821589 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c775a8-acfa-4bad-a4bf-a61143f7bff3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:40:04.821913 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c775a8-acfa-4bad-a4bf-a61143f7bff3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:04.823598 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c775a8-acfa-4bad-a4bf-a61143f7bff3 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:04.838007 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 983/1933] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:40:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:40:04.843323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a879088b-7d4d-4e7d-a898-3db12feb26f7 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:04.852924 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a879088b-7d4d-4e7d-a898-3db12feb26f7 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:04.858715 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cebe174-496e-43f3-b6d3-ac6e9584035e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:04.859541 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cebe174-496e-43f3-b6d3-ac6e9584035e admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:04.859688 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cebe174-496e-43f3-b6d3-ac6e9584035e admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:04.859973 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cebe174-496e-43f3-b6d3-ac6e9584035e admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:04.860691 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cebe174-496e-43f3-b6d3-ac6e9584035e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:04.862814 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cebe174-496e-43f3-b6d3-ac6e9584035e admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:04.906964 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:40:05.036165 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 951/1934] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:40:04 2022] POST /identity/v3/projects => generated 377 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:40:05.046310 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c775a8-acfa-4bad-a4bf-a61143f7bff3 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:05.060460 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c775a8-acfa-4bad-a4bf-a61143f7bff3 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:05.142639 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be613364-e7b8-430b-8ece-fe11ddb659ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:05.143340 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be613364-e7b8-430b-8ece-fe11ddb659ab admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:05.143638 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be613364-e7b8-430b-8ece-fe11ddb659ab admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:05.143920 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be613364-e7b8-430b-8ece-fe11ddb659ab admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:05.145095 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be613364-e7b8-430b-8ece-fe11ddb659ab admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:05.146740 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be613364-e7b8-430b-8ece-fe11ddb659ab admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:05.303333 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 984/1935] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:40:05 2022] POST /identity/v3/users => generated 289 bytes in 261 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:40:05.307633 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cebe174-496e-43f3-b6d3-ac6e9584035e admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:05.322808 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cebe174-496e-43f3-b6d3-ac6e9584035e admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:05.330379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7eb286-cfb2-42e9-ba7d-9961f669c3ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:05.331432 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7eb286-cfb2-42e9-ba7d-9961f669c3ee admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:05.331959 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7eb286-cfb2-42e9-ba7d-9961f669c3ee admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:05.332498 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7eb286-cfb2-42e9-ba7d-9961f669c3ee admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:05.333910 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b7eb286-cfb2-42e9-ba7d-9961f669c3ee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:05.336265 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b7eb286-cfb2-42e9-ba7d-9961f669c3ee admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:05.358615 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 952/1936] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:40:05 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 12 17:40:05.363563 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be613364-e7b8-430b-8ece-fe11ddb659ab admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:05.383292 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be613364-e7b8-430b-8ece-fe11ddb659ab admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:05.391795 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870d86cb-7f6a-4382-98ec-4cb465c4e6d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:05.393564 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870d86cb-7f6a-4382-98ec-4cb465c4e6d8 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:05.393859 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870d86cb-7f6a-4382-98ec-4cb465c4e6d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:05.394149 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870d86cb-7f6a-4382-98ec-4cb465c4e6d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:05.395718 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870d86cb-7f6a-4382-98ec-4cb465c4e6d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:05.404532 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870d86cb-7f6a-4382-98ec-4cb465c4e6d8 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:05.429309 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 985/1937] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:40:05 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:40:05.435534 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7eb286-cfb2-42e9-ba7d-9961f669c3ee admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:05.449544 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7eb286-cfb2-42e9-ba7d-9961f669c3ee admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:05.464804 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a150bf-6f11-4aef-a06a-499a9462ab23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:05.464804 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a150bf-6f11-4aef-a06a-499a9462ab23 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:05.464804 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a150bf-6f11-4aef-a06a-499a9462ab23 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:05.464804 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a150bf-6f11-4aef-a06a-499a9462ab23 admin admin] PATH_INFO: `/v3/projects/f075a640dab64ca982a28cf07caa6399/users/c418c6af46934596af3b4133a35ba1b4/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:05.496382 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a150bf-6f11-4aef-a06a-499a9462ab23 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f075a640dab64ca982a28cf07caa6399, user_id=c418c6af46934596af3b4133a35ba1b4, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:05.507554 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a150bf-6f11-4aef-a06a-499a9462ab23 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:05.640482 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 953/1938] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:40:05 2022] PUT /identity/v3/projects/f075a640dab64ca982a28cf07caa6399/users/c418c6af46934596af3b4133a35ba1b4/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:40:05.661021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebe86bf-546b-4bee-96d0-b4cb9a00f46d None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:05.661879 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebe86bf-546b-4bee-96d0-b4cb9a00f46d None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:05.662028 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebe86bf-546b-4bee-96d0-b4cb9a00f46d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:05.707413 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-2ebe86bf-546b-4bee-96d0-b4cb9a00f46d None None] MFA Rules not processed for user `c418c6af46934596af3b4133a35ba1b4`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:40:05.709609 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-2ebe86bf-546b-4bee-96d0-b4cb9a00f46d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:05.727255 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a150bf-6f11-4aef-a06a-499a9462ab23 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:05.730273 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-65a150bf-6f11-4aef-a06a-499a9462ab23 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:05.758655 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:40:05.838891 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 986/1939] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:40:05 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:40:05.867097 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebe86bf-546b-4bee-96d0-b4cb9a00f46d None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:05.870645 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-2ebe86bf-546b-4bee-96d0-b4cb9a00f46d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:05.894287 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a150bf-6f11-4aef-a06a-499a9462ab23 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:05.906067 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86f894b7-97e9-4263-8c41-f65b58eb97a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:05.907783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f894b7-97e9-4263-8c41-f65b58eb97a9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:05.908406 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f894b7-97e9-4263-8c41-f65b58eb97a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:05.909118 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f894b7-97e9-4263-8c41-f65b58eb97a9 admin admin] PATH_INFO: `/v3/projects/8f5604ce279346d39eba7e3db6a61e4f` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:05.911661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86f894b7-97e9-4263-8c41-f65b58eb97a9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8f5604ce279346d39eba7e3db6a61e4f)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:05.914890 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86f894b7-97e9-4263-8c41-f65b58eb97a9 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:05.992212 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebe86bf-546b-4bee-96d0-b4cb9a00f46d None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:05.999584 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ffaa2f1-a6ee-44d4-b30b-2508251e5957 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:06.000735 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffaa2f1-a6ee-44d4-b30b-2508251e5957 service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:06.001234 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffaa2f1-a6ee-44d4-b30b-2508251e5957 service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:06.001748 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ffaa2f1-a6ee-44d4-b30b-2508251e5957 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:06.014135 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ffaa2f1-a6ee-44d4-b30b-2508251e5957 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:06.015661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ffaa2f1-a6ee-44d4-b30b-2508251e5957 service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:06.026152 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 987/1940] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:40:05 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:40:06.099874 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-86f894b7-97e9-4263-8c41-f65b58eb97a9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8f5604ce279346d39eba7e3db6a61e4f', 'request_id': 'req-86f894b7-97e9-4263-8c41-f65b58eb97a9'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:06.108635 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:40:06.110770 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-86f894b7-97e9-4263-8c41-f65b58eb97a9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8f5604ce279346d39eba7e3db6a61e4f', 'request_id': 'req-86f894b7-97e9-4263-8c41-f65b58eb97a9'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:06.123361 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-86f894b7-97e9-4263-8c41-f65b58eb97a9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8f5604ce279346d39eba7e3db6a61e4f', 'request_id': 'req-86f894b7-97e9-4263-8c41-f65b58eb97a9'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:06.195085 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-86f894b7-97e9-4263-8c41-f65b58eb97a9 admin admin] The token cache is being invalidate because project 8f5604ce279346d39eba7e3db6a61e4f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:40:06.195498 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-86f894b7-97e9-4263-8c41-f65b58eb97a9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:06.205999 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-86f894b7-97e9-4263-8c41-f65b58eb97a9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:06.223165 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 954/1941] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:40:05 2022] DELETE /identity/v3/projects/8f5604ce279346d39eba7e3db6a61e4f => generated 0 bytes in 485 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:40:06.224405 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9050cbe1-a31f-4716-a01c-28d2e0c087fe None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:06.224693 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9050cbe1-a31f-4716-a01c-28d2e0c087fe None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:06.225148 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9050cbe1-a31f-4716-a01c-28d2e0c087fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:06.250849 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-9050cbe1-a31f-4716-a01c-28d2e0c087fe None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:40:06.252228 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-9050cbe1-a31f-4716-a01c-28d2e0c087fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:06.339534 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 988/1942] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:40:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:40:06.348507 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e9e3c7-2819-4ffd-bc38-d685e8359ac7 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:06.349024 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e9e3c7-2819-4ffd-bc38-d685e8359ac7 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:06.349416 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e9e3c7-2819-4ffd-bc38-d685e8359ac7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:06.372637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-a2e9e3c7-2819-4ffd-bc38-d685e8359ac7 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:40:06.374900 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-a2e9e3c7-2819-4ffd-bc38-d685e8359ac7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:06.384147 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 955/1943] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:40:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:40:06.388278 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9050cbe1-a31f-4716-a01c-28d2e0c087fe None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:06.399826 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9050cbe1-a31f-4716-a01c-28d2e0c087fe None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:06.405411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f357ca-518b-42ca-9f1c-db92a58c1403 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:06.409307 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f357ca-518b-42ca-9f1c-db92a58c1403 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:06.409307 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f357ca-518b-42ca-9f1c-db92a58c1403 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:06.409307 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f357ca-518b-42ca-9f1c-db92a58c1403 admin admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:06.409307 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f357ca-518b-42ca-9f1c-db92a58c1403 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:40:06.409307 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f357ca-518b-42ca-9f1c-db92a58c1403 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:06.410402 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f357ca-518b-42ca-9f1c-db92a58c1403 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:06.425248 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 989/1944] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:40:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:40:06.429499 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e9e3c7-2819-4ffd-bc38-d685e8359ac7 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:06.441689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e9e3c7-2819-4ffd-bc38-d685e8359ac7 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:06.446985 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb230b2-ef92-4658-ad3b-01edd791f029 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:06.449736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb230b2-ef92-4658-ad3b-01edd791f029 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:06.449736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb230b2-ef92-4658-ad3b-01edd791f029 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:06.449736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb230b2-ef92-4658-ad3b-01edd791f029 admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:06.449736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb230b2-ef92-4658-ad3b-01edd791f029 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:06.452091 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb230b2-ef92-4658-ad3b-01edd791f029 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:06.585145 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 956/1945] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:40:06 2022] POST /identity/v3/projects => generated 389 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:40:06.586579 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f357ca-518b-42ca-9f1c-db92a58c1403 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:06.598841 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f357ca-518b-42ca-9f1c-db92a58c1403 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:06.687600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fffc4f7-69c6-4ac6-b7e9-4795e453a87c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:06.688347 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fffc4f7-69c6-4ac6-b7e9-4795e453a87c admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:06.688933 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fffc4f7-69c6-4ac6-b7e9-4795e453a87c admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:06.689122 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fffc4f7-69c6-4ac6-b7e9-4795e453a87c admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:06.689831 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fffc4f7-69c6-4ac6-b7e9-4795e453a87c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:06.693103 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fffc4f7-69c6-4ac6-b7e9-4795e453a87c admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:06.803027 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb230b2-ef92-4658-ad3b-01edd791f029 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:06.805390 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 990/1946] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:40:06 2022] POST /identity/v3/users => generated 295 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:40:06.815803 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb230b2-ef92-4658-ad3b-01edd791f029 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:06.821232 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e7bd53-7d86-420e-8a01-bef13b8da9ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:06.829090 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e7bd53-7d86-420e-8a01-bef13b8da9ae admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:06.829090 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e7bd53-7d86-420e-8a01-bef13b8da9ae admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:06.829090 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29e7bd53-7d86-420e-8a01-bef13b8da9ae admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:06.829090 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e7bd53-7d86-420e-8a01-bef13b8da9ae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:06.829090 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29e7bd53-7d86-420e-8a01-bef13b8da9ae admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:06.839587 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 957/1947] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:40:06 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:40:06.844506 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fffc4f7-69c6-4ac6-b7e9-4795e453a87c admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:06.861815 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fffc4f7-69c6-4ac6-b7e9-4795e453a87c admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:06.866926 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d24ae3d-36f1-4069-9bd0-badcc071bf58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:06.867673 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d24ae3d-36f1-4069-9bd0-badcc071bf58 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:06.867949 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d24ae3d-36f1-4069-9bd0-badcc071bf58 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:06.868233 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d24ae3d-36f1-4069-9bd0-badcc071bf58 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:06.869160 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d24ae3d-36f1-4069-9bd0-badcc071bf58 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:06.870839 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d24ae3d-36f1-4069-9bd0-badcc071bf58 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:06.892998 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 991/1948] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:40:06 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 12 17:40:06.894481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e7bd53-7d86-420e-8a01-bef13b8da9ae admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:06.907791 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29e7bd53-7d86-420e-8a01-bef13b8da9ae admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:06.915081 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0d32bd-8489-4968-84c0-09da11d79a9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:06.916376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0d32bd-8489-4968-84c0-09da11d79a9a admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:06.916870 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0d32bd-8489-4968-84c0-09da11d79a9a admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:06.917308 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0d32bd-8489-4968-84c0-09da11d79a9a admin admin] PATH_INFO: `/v3/projects/580553bc0e4b4c9cba679c935643fe1d/users/22682c4345c248d1aee77a6905472934/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:06.948323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca0d32bd-8489-4968-84c0-09da11d79a9a admin admin] RBAC: Authorizing `identity:create_grant(project_id=580553bc0e4b4c9cba679c935643fe1d, user_id=22682c4345c248d1aee77a6905472934, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:06.952861 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca0d32bd-8489-4968-84c0-09da11d79a9a admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:07.097414 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 958/1949] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:40:06 2022] PUT /identity/v3/projects/580553bc0e4b4c9cba679c935643fe1d/users/22682c4345c248d1aee77a6905472934/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:40:07.109328 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f30fe77-1ce9-408f-8674-801c5d08c946 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:07.109629 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f30fe77-1ce9-408f-8674-801c5d08c946 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:07.109924 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f30fe77-1ce9-408f-8674-801c5d08c946 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:07.164166 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-6f30fe77-1ce9-408f-8674-801c5d08c946 None None] MFA Rules not processed for user `22682c4345c248d1aee77a6905472934`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:40:07.165617 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-6f30fe77-1ce9-408f-8674-801c5d08c946 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:07.250022 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 992/1950] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:40:07 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:40:07.261630 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0d32bd-8489-4968-84c0-09da11d79a9a admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:07.264876 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-ca0d32bd-8489-4968-84c0-09da11d79a9a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:07.330272 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0d32bd-8489-4968-84c0-09da11d79a9a admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:07.334407 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebe9e54e-9013-4fc8-9bc7-2299fdae15e2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:07.335104 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe9e54e-9013-4fc8-9bc7-2299fdae15e2 service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:07.335413 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe9e54e-9013-4fc8-9bc7-2299fdae15e2 service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:07.335679 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebe9e54e-9013-4fc8-9bc7-2299fdae15e2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:07.346632 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe9e54e-9013-4fc8-9bc7-2299fdae15e2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:07.348514 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebe9e54e-9013-4fc8-9bc7-2299fdae15e2 service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:07.394400 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 959/1951] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:40:07 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:40:08.231887 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:40:08.884928 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:40:10.468576 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:40:11.335863 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:40:12.021714 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:40:12.374955 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:40:12.654819 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:40:12.963949 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:40:13.362391 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:40:13.420655 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:40:13.786242 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:DELETE=3,keystone:SELECT=93,keystone:INSERT=4 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:40:13.998905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:40:15.233042 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:40:25.523144 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=7 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:40:25.906056 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=13 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:40:44.331496 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f30fe77-1ce9-408f-8674-801c5d08c946 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:44.440673 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f30fe77-1ce9-408f-8674-801c5d08c946 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:44.445281 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2da789-023e-4341-90f3-9a0386490395 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:44.446203 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2da789-023e-4341-90f3-9a0386490395 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:44.446524 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2da789-023e-4341-90f3-9a0386490395 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:44.446809 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2da789-023e-4341-90f3-9a0386490395 admin admin] PATH_INFO: `/v3/users/22682c4345c248d1aee77a6905472934` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:44.448167 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2da789-023e-4341-90f3-9a0386490395 admin admin] RBAC: Authorizing `identity:delete_user(user_id=22682c4345c248d1aee77a6905472934)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:44.449970 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2da789-023e-4341-90f3-9a0386490395 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:44.649122 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ba2da789-023e-4341-90f3-9a0386490395 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '22682c4345c248d1aee77a6905472934'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:44.650105 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ba2da789-023e-4341-90f3-9a0386490395 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '22682c4345c248d1aee77a6905472934'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:44.651205 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ba2da789-023e-4341-90f3-9a0386490395 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '22682c4345c248d1aee77a6905472934'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:44.668688 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ba2da789-023e-4341-90f3-9a0386490395 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '22682c4345c248d1aee77a6905472934'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:44.683337 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ba2da789-023e-4341-90f3-9a0386490395 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '22682c4345c248d1aee77a6905472934'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:44.780180 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 993/1952] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:40:44 2022] DELETE /identity/v3/users/22682c4345c248d1aee77a6905472934 => generated 0 bytes in 449 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:40:44.785905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c444a976-bc3b-49e5-a296-e0ef4cdff994 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:44.786888 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c444a976-bc3b-49e5-a296-e0ef4cdff994 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:44.786991 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c444a976-bc3b-49e5-a296-e0ef4cdff994 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:44.813145 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-c444a976-bc3b-49e5-a296-e0ef4cdff994 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:40:44.814841 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-c444a976-bc3b-49e5-a296-e0ef4cdff994 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:44.906694 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 960/1953] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:40:44 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:40:44.940791 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2da789-023e-4341-90f3-9a0386490395 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:44.942384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-ba2da789-023e-4341-90f3-9a0386490395 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:45.028864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2da789-023e-4341-90f3-9a0386490395 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:45.038914 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08dd9a2d-cad1-485f-997a-317e3de79da0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:45.040077 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08dd9a2d-cad1-485f-997a-317e3de79da0 service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:45.040543 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08dd9a2d-cad1-485f-997a-317e3de79da0 service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:45.041189 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08dd9a2d-cad1-485f-997a-317e3de79da0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:45.058966 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08dd9a2d-cad1-485f-997a-317e3de79da0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:45.063894 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08dd9a2d-cad1-485f-997a-317e3de79da0 service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:45.079291 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 994/1954] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:40:44 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:40:45.792430 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c444a976-bc3b-49e5-a296-e0ef4cdff994 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:45.796635 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-c444a976-bc3b-49e5-a296-e0ef4cdff994 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:45.815196 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c444a976-bc3b-49e5-a296-e0ef4cdff994 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:45.820827 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5a4d31b-e135-4d47-94ff-5f4ffa30570e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:45.822029 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a4d31b-e135-4d47-94ff-5f4ffa30570e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:45.822340 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a4d31b-e135-4d47-94ff-5f4ffa30570e admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:45.822688 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5a4d31b-e135-4d47-94ff-5f4ffa30570e admin admin] PATH_INFO: `/v3/projects/580553bc0e4b4c9cba679c935643fe1d` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:45.824190 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a4d31b-e135-4d47-94ff-5f4ffa30570e admin admin] RBAC: Authorizing `identity:delete_project(project_id=580553bc0e4b4c9cba679c935643fe1d)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:45.827889 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5a4d31b-e135-4d47-94ff-5f4ffa30570e admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:45.963438 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-a5a4d31b-e135-4d47-94ff-5f4ffa30570e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '580553bc0e4b4c9cba679c935643fe1d', 'request_id': 'req-a5a4d31b-e135-4d47-94ff-5f4ffa30570e'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:45.973259 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-a5a4d31b-e135-4d47-94ff-5f4ffa30570e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '580553bc0e4b4c9cba679c935643fe1d', 'request_id': 'req-a5a4d31b-e135-4d47-94ff-5f4ffa30570e'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:45.980453 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-a5a4d31b-e135-4d47-94ff-5f4ffa30570e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '580553bc0e4b4c9cba679c935643fe1d', 'request_id': 'req-a5a4d31b-e135-4d47-94ff-5f4ffa30570e'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:46.072193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-a5a4d31b-e135-4d47-94ff-5f4ffa30570e admin admin] The token cache is being invalidate because project 580553bc0e4b4c9cba679c935643fe1d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:40:46.072443 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-a5a4d31b-e135-4d47-94ff-5f4ffa30570e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:46.073159 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-a5a4d31b-e135-4d47-94ff-5f4ffa30570e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:46.074956 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 961/1955] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:40:45 2022] DELETE /identity/v3/projects/580553bc0e4b4c9cba679c935643fe1d => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:40:46.081039 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb50f2d-86a2-4a1f-bf1d-92a3add0be41 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:46.081992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb50f2d-86a2-4a1f-bf1d-92a3add0be41 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:46.081992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb50f2d-86a2-4a1f-bf1d-92a3add0be41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:46.111409 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-0cb50f2d-86a2-4a1f-bf1d-92a3add0be41 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:40:46.112648 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-0cb50f2d-86a2-4a1f-bf1d-92a3add0be41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:46.134294 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:40:46.253419 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 995/1956] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:40:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:40:46.261501 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438656c9-d391-47b7-8119-1ebe53e4be13 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:46.261795 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438656c9-d391-47b7-8119-1ebe53e4be13 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:46.262083 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438656c9-d391-47b7-8119-1ebe53e4be13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:46.286137 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-438656c9-d391-47b7-8119-1ebe53e4be13 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:40:46.287517 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-438656c9-d391-47b7-8119-1ebe53e4be13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:46.304492 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb50f2d-86a2-4a1f-bf1d-92a3add0be41 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:46.318032 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 962/1957] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:40:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:40:46.318178 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb50f2d-86a2-4a1f-bf1d-92a3add0be41 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:46.319165 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6607e0-69d5-4b8b-a140-75f31b4ffbbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:46.320053 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6607e0-69d5-4b8b-a140-75f31b4ffbbe admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:46.321495 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6607e0-69d5-4b8b-a140-75f31b4ffbbe admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:46.321871 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6607e0-69d5-4b8b-a140-75f31b4ffbbe admin admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:46.322812 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6607e0-69d5-4b8b-a140-75f31b4ffbbe admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:40:46.323205 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6607e0-69d5-4b8b-a140-75f31b4ffbbe admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:46.325764 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6607e0-69d5-4b8b-a140-75f31b4ffbbe admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:46.346298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 996/1958] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:40:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:40:46.347636 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438656c9-d391-47b7-8119-1ebe53e4be13 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:46.359724 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438656c9-d391-47b7-8119-1ebe53e4be13 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:46.364963 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30666b96-6f07-4c3d-b0cf-147c458fc3ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:46.365704 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30666b96-6f07-4c3d-b0cf-147c458fc3ea admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:46.366172 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30666b96-6f07-4c3d-b0cf-147c458fc3ea admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:46.366559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30666b96-6f07-4c3d-b0cf-147c458fc3ea admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:46.367263 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30666b96-6f07-4c3d-b0cf-147c458fc3ea admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:46.369174 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30666b96-6f07-4c3d-b0cf-147c458fc3ea admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:46.517533 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 963/1959] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:40:46 2022] POST /identity/v3/projects => generated 395 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:40:46.523603 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6607e0-69d5-4b8b-a140-75f31b4ffbbe admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:46.537834 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6607e0-69d5-4b8b-a140-75f31b4ffbbe admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:46.620574 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706f55d6-bb6c-47e9-a328-54f9ee68e122 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:46.621781 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706f55d6-bb6c-47e9-a328-54f9ee68e122 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:46.622222 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706f55d6-bb6c-47e9-a328-54f9ee68e122 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:46.622716 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706f55d6-bb6c-47e9-a328-54f9ee68e122 admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:46.626741 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706f55d6-bb6c-47e9-a328-54f9ee68e122 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:46.629739 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706f55d6-bb6c-47e9-a328-54f9ee68e122 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:46.733940 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30666b96-6f07-4c3d-b0cf-147c458fc3ea admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:46.734205 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 997/1960] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:40:46 2022] POST /identity/v3/users => generated 298 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:40:46.762401 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30666b96-6f07-4c3d-b0cf-147c458fc3ea admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:46.782202 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773f31e1-2951-4898-83a5-2bc86182e0ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:46.782488 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773f31e1-2951-4898-83a5-2bc86182e0ee admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:46.783465 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773f31e1-2951-4898-83a5-2bc86182e0ee admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:46.783465 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773f31e1-2951-4898-83a5-2bc86182e0ee admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:46.787972 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-773f31e1-2951-4898-83a5-2bc86182e0ee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:46.791063 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-773f31e1-2951-4898-83a5-2bc86182e0ee admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:46.815890 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706f55d6-bb6c-47e9-a328-54f9ee68e122 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:46.816501 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 964/1961] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:40:46 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:40:46.833554 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706f55d6-bb6c-47e9-a328-54f9ee68e122 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:46.841017 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24dc204-3b3e-41f2-955e-baf3df591dba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:46.843348 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24dc204-3b3e-41f2-955e-baf3df591dba admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:46.843852 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24dc204-3b3e-41f2-955e-baf3df591dba admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:46.847930 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24dc204-3b3e-41f2-955e-baf3df591dba admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:46.847930 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d24dc204-3b3e-41f2-955e-baf3df591dba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:46.848260 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d24dc204-3b3e-41f2-955e-baf3df591dba admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:46.860777 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 998/1962] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:40:46 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:40:46.865554 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773f31e1-2951-4898-83a5-2bc86182e0ee admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:46.877500 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773f31e1-2951-4898-83a5-2bc86182e0ee admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:46.885088 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f237cd06-345d-4a8d-b291-086ef4ef4905 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:46.886747 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f237cd06-345d-4a8d-b291-086ef4ef4905 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:46.887260 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f237cd06-345d-4a8d-b291-086ef4ef4905 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:46.888011 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f237cd06-345d-4a8d-b291-086ef4ef4905 admin admin] PATH_INFO: `/v3/projects/c8c850e77a524184a741b0857200d0a5/users/e81b12a867af4d088a67a001e58ac9f9/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:46.915744 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f237cd06-345d-4a8d-b291-086ef4ef4905 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c8c850e77a524184a741b0857200d0a5, user_id=e81b12a867af4d088a67a001e58ac9f9, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:46.921714 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f237cd06-345d-4a8d-b291-086ef4ef4905 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:47.025257 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 965/1963] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:40:46 2022] PUT /identity/v3/projects/c8c850e77a524184a741b0857200d0a5/users/e81b12a867af4d088a67a001e58ac9f9/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:40:47.037721 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad3102f-90a7-46cd-a6e3-d5023aaaadbb None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:47.038021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad3102f-90a7-46cd-a6e3-d5023aaaadbb None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:47.038311 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad3102f-90a7-46cd-a6e3-d5023aaaadbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:47.066665 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-4ad3102f-90a7-46cd-a6e3-d5023aaaadbb None None] MFA Rules not processed for user `e81b12a867af4d088a67a001e58ac9f9`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:40:47.067907 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-4ad3102f-90a7-46cd-a6e3-d5023aaaadbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:47.137868 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 999/1964] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:40:47 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:40:47.186621 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f237cd06-345d-4a8d-b291-086ef4ef4905 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:47.186621 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-f237cd06-345d-4a8d-b291-086ef4ef4905 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:47.186621 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:40:47.212930 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f237cd06-345d-4a8d-b291-086ef4ef4905 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:47.217127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55824e2e-1e60-4f74-b31c-78ae20dc5910 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:47.217823 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55824e2e-1e60-4f74-b31c-78ae20dc5910 service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:47.220617 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55824e2e-1e60-4f74-b31c-78ae20dc5910 service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:47.220617 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55824e2e-1e60-4f74-b31c-78ae20dc5910 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:47.227762 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55824e2e-1e60-4f74-b31c-78ae20dc5910 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:47.229287 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55824e2e-1e60-4f74-b31c-78ae20dc5910 service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:47.239365 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 966/1965] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:40:47 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:40:49.559774 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b951a9-4aa8-4a3e-8271-43a592416f37 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:49.560448 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b951a9-4aa8-4a3e-8271-43a592416f37 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:49.560448 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b951a9-4aa8-4a3e-8271-43a592416f37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:49.579839 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-01b951a9-4aa8-4a3e-8271-43a592416f37 None None] MFA Rules not processed for user `2c4deca7942842759f795ab5299b44a3`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:40:49.580911 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-01b951a9-4aa8-4a3e-8271-43a592416f37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:49.593765 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:40:49.670516 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1000/1966] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:40:49 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:40:54.084410 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55824e2e-1e60-4f74-b31c-78ae20dc5910 service cinder] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:54.086574 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-55824e2e-1e60-4f74-b31c-78ae20dc5910 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:54.196381 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55824e2e-1e60-4f74-b31c-78ae20dc5910 service cinder] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:54.201563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d822f5-c422-4acd-998e-26b90f0133f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:54.203114 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d822f5-c422-4acd-998e-26b90f0133f3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:54.203409 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d822f5-c422-4acd-998e-26b90f0133f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:54.203711 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d822f5-c422-4acd-998e-26b90f0133f3 admin admin] PATH_INFO: `/v3/users/c418c6af46934596af3b4133a35ba1b4` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:54.205250 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d822f5-c422-4acd-998e-26b90f0133f3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c418c6af46934596af3b4133a35ba1b4)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:54.207332 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d822f5-c422-4acd-998e-26b90f0133f3 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:54.507154 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-d2d822f5-c422-4acd-998e-26b90f0133f3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c418c6af46934596af3b4133a35ba1b4'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:54.581543 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-d2d822f5-c422-4acd-998e-26b90f0133f3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c418c6af46934596af3b4133a35ba1b4'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:54.582812 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-d2d822f5-c422-4acd-998e-26b90f0133f3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c418c6af46934596af3b4133a35ba1b4'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:54.599177 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-d2d822f5-c422-4acd-998e-26b90f0133f3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c418c6af46934596af3b4133a35ba1b4'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:54.626892 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-d2d822f5-c422-4acd-998e-26b90f0133f3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c418c6af46934596af3b4133a35ba1b4'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:54.641474 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 967/1967] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:40:54 2022] DELETE /identity/v3/users/c418c6af46934596af3b4133a35ba1b4 => generated 0 bytes in 558 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:40:54.647638 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9256b70-469f-4615-838a-2e627de02833 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:54.648320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9256b70-469f-4615-838a-2e627de02833 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:54.648320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9256b70-469f-4615-838a-2e627de02833 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:54.674760 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-c9256b70-469f-4615-838a-2e627de02833 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:40:54.674760 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-c9256b70-469f-4615-838a-2e627de02833 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:54.796011 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1001/1968] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:40:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:40:54.807726 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d822f5-c422-4acd-998e-26b90f0133f3 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:54.809403 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-d2d822f5-c422-4acd-998e-26b90f0133f3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:54.901053 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:40:54.931582 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d822f5-c422-4acd-998e-26b90f0133f3 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:54.942214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d617f380-37f6-4b99-9278-a2237fc384e6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:54.943525 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d617f380-37f6-4b99-9278-a2237fc384e6 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:54.944660 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d617f380-37f6-4b99-9278-a2237fc384e6 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:54.945190 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d617f380-37f6-4b99-9278-a2237fc384e6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:54.967258 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d617f380-37f6-4b99-9278-a2237fc384e6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:54.970106 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d617f380-37f6-4b99-9278-a2237fc384e6 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:54.989783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 968/1969] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:40:54 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 183 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:40:55.651692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9256b70-469f-4615-838a-2e627de02833 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:55.654103 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-c9256b70-469f-4615-838a-2e627de02833 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:55.701625 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9256b70-469f-4615-838a-2e627de02833 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:55.707534 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a3be688-b826-4bfd-bc35-84c59de26309 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:55.708587 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3be688-b826-4bfd-bc35-84c59de26309 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:55.708971 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3be688-b826-4bfd-bc35-84c59de26309 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:55.709361 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3be688-b826-4bfd-bc35-84c59de26309 admin admin] PATH_INFO: `/v3/projects/f075a640dab64ca982a28cf07caa6399` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:55.710608 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a3be688-b826-4bfd-bc35-84c59de26309 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f075a640dab64ca982a28cf07caa6399)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:55.713474 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a3be688-b826-4bfd-bc35-84c59de26309 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:55.860736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-7a3be688-b826-4bfd-bc35-84c59de26309 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f075a640dab64ca982a28cf07caa6399', 'request_id': 'req-7a3be688-b826-4bfd-bc35-84c59de26309'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:55.873620 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-7a3be688-b826-4bfd-bc35-84c59de26309 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f075a640dab64ca982a28cf07caa6399', 'request_id': 'req-7a3be688-b826-4bfd-bc35-84c59de26309'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:55.979937 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-7a3be688-b826-4bfd-bc35-84c59de26309 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f075a640dab64ca982a28cf07caa6399', 'request_id': 'req-7a3be688-b826-4bfd-bc35-84c59de26309'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:55.995042 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-7a3be688-b826-4bfd-bc35-84c59de26309 admin admin] The token cache is being invalidate because project f075a640dab64ca982a28cf07caa6399 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:40:55.995327 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-7a3be688-b826-4bfd-bc35-84c59de26309 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:55.996304 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-7a3be688-b826-4bfd-bc35-84c59de26309 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:40:56.003014 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1002/1970] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:40:55 2022] DELETE /identity/v3/projects/f075a640dab64ca982a28cf07caa6399 => generated 0 bytes in 348 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:40:56.004973 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba515e5-106b-4eba-a02f-bf1fdaef8750 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:56.005723 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba515e5-106b-4eba-a02f-bf1fdaef8750 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:56.005934 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba515e5-106b-4eba-a02f-bf1fdaef8750 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:56.025641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-4ba515e5-106b-4eba-a02f-bf1fdaef8750 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:40:56.026919 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-4ba515e5-106b-4eba-a02f-bf1fdaef8750 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:56.134951 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 969/1971] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:40:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:40:56.146602 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c95ab99-2ff7-4621-8329-99fd6ee1c56e None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:56.147535 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c95ab99-2ff7-4621-8329-99fd6ee1c56e None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:56.148358 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c95ab99-2ff7-4621-8329-99fd6ee1c56e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:56.195274 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-1c95ab99-2ff7-4621-8329-99fd6ee1c56e None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:40:56.197700 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-1c95ab99-2ff7-4621-8329-99fd6ee1c56e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:56.216471 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1003/1972] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:40:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 72 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:40:56.234654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba515e5-106b-4eba-a02f-bf1fdaef8750 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:56.234654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ba515e5-106b-4eba-a02f-bf1fdaef8750 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:56.240828 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9c72ed-42ba-45d4-8d5c-9fae9117cbc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:56.241767 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9c72ed-42ba-45d4-8d5c-9fae9117cbc5 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:56.242164 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9c72ed-42ba-45d4-8d5c-9fae9117cbc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:56.242744 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9c72ed-42ba-45d4-8d5c-9fae9117cbc5 admin admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:56.243779 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9c72ed-42ba-45d4-8d5c-9fae9117cbc5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:40:56.244204 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9c72ed-42ba-45d4-8d5c-9fae9117cbc5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:56.246191 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9c72ed-42ba-45d4-8d5c-9fae9117cbc5 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:56.265690 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 970/1973] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:40:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:40:56.284212 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c95ab99-2ff7-4621-8329-99fd6ee1c56e None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:56.299485 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c95ab99-2ff7-4621-8329-99fd6ee1c56e None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:56.307161 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07786db-8bc2-40d8-8dd0-c2c25f3e1730 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:56.308160 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07786db-8bc2-40d8-8dd0-c2c25f3e1730 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:56.308841 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07786db-8bc2-40d8-8dd0-c2c25f3e1730 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:56.309537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07786db-8bc2-40d8-8dd0-c2c25f3e1730 admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:56.310619 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f07786db-8bc2-40d8-8dd0-c2c25f3e1730 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:56.313803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f07786db-8bc2-40d8-8dd0-c2c25f3e1730 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:56.463664 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1004/1974] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:40:56 2022] POST /identity/v3/projects => generated 369 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:40:56.470648 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9c72ed-42ba-45d4-8d5c-9fae9117cbc5 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:56.494514 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9c72ed-42ba-45d4-8d5c-9fae9117cbc5 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:56.591163 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213bd034-0da3-4989-93c0-72baf0e09cbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:56.591904 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213bd034-0da3-4989-93c0-72baf0e09cbf admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:56.592659 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213bd034-0da3-4989-93c0-72baf0e09cbf admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:56.592659 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213bd034-0da3-4989-93c0-72baf0e09cbf admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:56.593410 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-213bd034-0da3-4989-93c0-72baf0e09cbf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:56.595681 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-213bd034-0da3-4989-93c0-72baf0e09cbf admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:57.001432 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 971/1975] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:40:56 2022] POST /identity/v3/users => generated 285 bytes in 531 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:40:57.006173 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07786db-8bc2-40d8-8dd0-c2c25f3e1730 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:57.018634 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07786db-8bc2-40d8-8dd0-c2c25f3e1730 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:57.023694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db7c0d4-1413-4d1e-ac3e-87e13540cf26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:57.024495 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db7c0d4-1413-4d1e-ac3e-87e13540cf26 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:57.024844 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db7c0d4-1413-4d1e-ac3e-87e13540cf26 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:57.025193 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db7c0d4-1413-4d1e-ac3e-87e13540cf26 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:57.026074 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db7c0d4-1413-4d1e-ac3e-87e13540cf26 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:57.027779 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db7c0d4-1413-4d1e-ac3e-87e13540cf26 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:57.040978 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1005/1976] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:40:57 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:40:57.045260 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213bd034-0da3-4989-93c0-72baf0e09cbf admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:57.055805 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213bd034-0da3-4989-93c0-72baf0e09cbf admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:57.061499 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53d95cb-7c78-4d76-96fa-21cc0f8645af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:57.063024 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53d95cb-7c78-4d76-96fa-21cc0f8645af admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:57.063475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53d95cb-7c78-4d76-96fa-21cc0f8645af admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:57.063774 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53d95cb-7c78-4d76-96fa-21cc0f8645af admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:57.064807 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53d95cb-7c78-4d76-96fa-21cc0f8645af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:57.066465 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53d95cb-7c78-4d76-96fa-21cc0f8645af admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:57.078506 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 972/1977] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:40:57 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 12 17:40:57.084022 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db7c0d4-1413-4d1e-ac3e-87e13540cf26 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:57.094806 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db7c0d4-1413-4d1e-ac3e-87e13540cf26 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:57.099726 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd820b0-4f05-4c23-ac88-69da6d39fb05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:57.100615 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd820b0-4f05-4c23-ac88-69da6d39fb05 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:57.100901 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd820b0-4f05-4c23-ac88-69da6d39fb05 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:57.101175 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd820b0-4f05-4c23-ac88-69da6d39fb05 admin admin] PATH_INFO: `/v3/projects/f71cf15bea67452a9ce256e788bb3bfe/users/44ff65795bf646429b317820c8af4e3c/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:57.103589 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:40:57.118201 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd820b0-4f05-4c23-ac88-69da6d39fb05 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f71cf15bea67452a9ce256e788bb3bfe, user_id=44ff65795bf646429b317820c8af4e3c, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:57.123885 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd820b0-4f05-4c23-ac88-69da6d39fb05 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:57.277374 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1006/1978] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:40:57 2022] PUT /identity/v3/projects/f71cf15bea67452a9ce256e788bb3bfe/users/44ff65795bf646429b317820c8af4e3c/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:40:57.289152 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a3cae7-6565-42a6-89b1-6677483beecd None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:57.289659 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a3cae7-6565-42a6-89b1-6677483beecd None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:57.290094 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a3cae7-6565-42a6-89b1-6677483beecd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:57.334782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-17a3cae7-6565-42a6-89b1-6677483beecd None None] MFA Rules not processed for user `44ff65795bf646429b317820c8af4e3c`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:40:57.336654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-17a3cae7-6565-42a6-89b1-6677483beecd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:57.446811 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:40:57.501910 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 973/1979] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:40:57 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:40:57.522283 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd820b0-4f05-4c23-ac88-69da6d39fb05 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:57.526038 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-3dd820b0-4f05-4c23-ac88-69da6d39fb05 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:57.711301 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd820b0-4f05-4c23-ac88-69da6d39fb05 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:57.723161 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b0b876-4879-4594-aa63-306f05ff2769 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4018d94a1feb4d6dade88ee9663f9729', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:57.723946 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b0b876-4879-4594-aa63-306f05ff2769 service glance] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:57.724347 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b0b876-4879-4594-aa63-306f05ff2769 service glance] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:57.724754 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b0b876-4879-4594-aa63-306f05ff2769 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:57.754217 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b0b876-4879-4594-aa63-306f05ff2769 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:57.759716 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b0b876-4879-4594-aa63-306f05ff2769 service glance] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:40:57.801690 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1007/1980] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:40:57 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 280 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:40:59.742873 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a3cae7-6565-42a6-89b1-6677483beecd None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:40:59.745387 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-17a3cae7-6565-42a6-89b1-6677483beecd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:40:59.880148 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a3cae7-6565-42a6-89b1-6677483beecd None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:40:59.887747 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7616c27-ba0d-44f8-872f-0047466ae094 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:40:59.887747 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7616c27-ba0d-44f8-872f-0047466ae094 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:40:59.887747 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7616c27-ba0d-44f8-872f-0047466ae094 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:40:59.887747 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7616c27-ba0d-44f8-872f-0047466ae094 admin admin] PATH_INFO: `/v3/users/0a3fc03652ed4c4089b5fb61c088ce29` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:40:59.889783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7616c27-ba0d-44f8-872f-0047466ae094 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0a3fc03652ed4c4089b5fb61c088ce29)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:40:59.892690 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7616c27-ba0d-44f8-872f-0047466ae094 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:00.016711 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b0b876-4879-4594-aa63-306f05ff2769 service glance] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:00.018948 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-42b0b876-4879-4594-aa63-306f05ff2769 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:00.041201 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b0b876-4879-4594-aa63-306f05ff2769 service glance] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:00.047489 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08cbd091-e805-4f5f-a636-4643530e5f5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:00.048778 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cbd091-e805-4f5f-a636-4643530e5f5b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:00.049316 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cbd091-e805-4f5f-a636-4643530e5f5b admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:00.050678 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cbd091-e805-4f5f-a636-4643530e5f5b admin admin] PATH_INFO: `/v3/users/cd79a28d8f774c1fac44185df36cd24f` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:00.052169 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08cbd091-e805-4f5f-a636-4643530e5f5b admin admin] RBAC: Authorizing `identity:delete_user(user_id=cd79a28d8f774c1fac44185df36cd24f)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:00.054712 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08cbd091-e805-4f5f-a636-4643530e5f5b admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:00.155311 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-b7616c27-ba0d-44f8-872f-0047466ae094 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0a3fc03652ed4c4089b5fb61c088ce29'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:00.271214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-b7616c27-ba0d-44f8-872f-0047466ae094 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0a3fc03652ed4c4089b5fb61c088ce29'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:00.272000 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-b7616c27-ba0d-44f8-872f-0047466ae094 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0a3fc03652ed4c4089b5fb61c088ce29'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:00.280163 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-b7616c27-ba0d-44f8-872f-0047466ae094 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0a3fc03652ed4c4089b5fb61c088ce29'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:00.304943 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-08cbd091-e805-4f5f-a636-4643530e5f5b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cd79a28d8f774c1fac44185df36cd24f'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:00.305545 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-b7616c27-ba0d-44f8-872f-0047466ae094 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0a3fc03652ed4c4089b5fb61c088ce29'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:00.305914 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-08cbd091-e805-4f5f-a636-4643530e5f5b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cd79a28d8f774c1fac44185df36cd24f'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:00.306812 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-08cbd091-e805-4f5f-a636-4643530e5f5b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cd79a28d8f774c1fac44185df36cd24f'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:00.315381 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 974/1981] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:40:59 2022] DELETE /identity/v3/users/0a3fc03652ed4c4089b5fb61c088ce29 => generated 0 bytes in 573 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:00.319622 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973233a0-70c4-4e94-9927-a0810b28f0eb None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:00.319927 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973233a0-70c4-4e94-9927-a0810b28f0eb None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:00.320211 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973233a0-70c4-4e94-9927-a0810b28f0eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:00.322994 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-08cbd091-e805-4f5f-a636-4643530e5f5b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cd79a28d8f774c1fac44185df36cd24f'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:00.338662 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-08cbd091-e805-4f5f-a636-4643530e5f5b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cd79a28d8f774c1fac44185df36cd24f'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:00.341549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-973233a0-70c4-4e94-9927-a0810b28f0eb None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:00.343452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-973233a0-70c4-4e94-9927-a0810b28f0eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:00.423662 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:00.432989 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 975/1982] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:41:00.435351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973233a0-70c4-4e94-9927-a0810b28f0eb None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:00.436681 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-973233a0-70c4-4e94-9927-a0810b28f0eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:00.450132 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973233a0-70c4-4e94-9927-a0810b28f0eb None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:00.459552 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ceb5a29-0b83-4a8b-83bd-9d474fa97627 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:00.459552 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ceb5a29-0b83-4a8b-83bd-9d474fa97627 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:00.459552 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ceb5a29-0b83-4a8b-83bd-9d474fa97627 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:00.459552 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ceb5a29-0b83-4a8b-83bd-9d474fa97627 admin admin] PATH_INFO: `/v3/users/e81b12a867af4d088a67a001e58ac9f9` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:00.460502 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ceb5a29-0b83-4a8b-83bd-9d474fa97627 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e81b12a867af4d088a67a001e58ac9f9)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:00.462286 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ceb5a29-0b83-4a8b-83bd-9d474fa97627 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:00.493829 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1008/1983] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:41:00 2022] DELETE /identity/v3/users/cd79a28d8f774c1fac44185df36cd24f => generated 0 bytes in 479 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:41:00.495040 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08cbd091-e805-4f5f-a636-4643530e5f5b admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:00.496724 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-08cbd091-e805-4f5f-a636-4643530e5f5b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:00.561249 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:00.587986 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08cbd091-e805-4f5f-a636-4643530e5f5b admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:00.594421 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02aa96e8-5eb4-47d0-8f8e-9285e9aa72ba service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:00.595571 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02aa96e8-5eb4-47d0-8f8e-9285e9aa72ba service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:00.596112 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02aa96e8-5eb4-47d0-8f8e-9285e9aa72ba service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:00.596614 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02aa96e8-5eb4-47d0-8f8e-9285e9aa72ba service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:00.696988 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02aa96e8-5eb4-47d0-8f8e-9285e9aa72ba service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:00.698464 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02aa96e8-5eb4-47d0-8f8e-9285e9aa72ba service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:00.712563 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1009/1984] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:41:00 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 218 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:41:00.799780 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-9ceb5a29-0b83-4a8b-83bd-9d474fa97627 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e81b12a867af4d088a67a001e58ac9f9'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:00.830780 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02aa96e8-5eb4-47d0-8f8e-9285e9aa72ba service neutron] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:00.832975 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-02aa96e8-5eb4-47d0-8f8e-9285e9aa72ba service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:00.869283 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02aa96e8-5eb4-47d0-8f8e-9285e9aa72ba service neutron] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:00.877373 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1108031f-76de-4707-adb4-c52165dc5edb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:00.878262 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1108031f-76de-4707-adb4-c52165dc5edb admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:00.878597 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1108031f-76de-4707-adb4-c52165dc5edb admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:00.878889 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1108031f-76de-4707-adb4-c52165dc5edb admin admin] PATH_INFO: `/v3/projects/33a7a8fa558b472da92a52cdd63bdbad` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:00.880156 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1108031f-76de-4707-adb4-c52165dc5edb admin admin] RBAC: Authorizing `identity:delete_project(project_id=33a7a8fa558b472da92a52cdd63bdbad)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:00.882347 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1108031f-76de-4707-adb4-c52165dc5edb admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:00.883237 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-9ceb5a29-0b83-4a8b-83bd-9d474fa97627 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e81b12a867af4d088a67a001e58ac9f9'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:00.884043 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-9ceb5a29-0b83-4a8b-83bd-9d474fa97627 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e81b12a867af4d088a67a001e58ac9f9'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:00.897785 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-9ceb5a29-0b83-4a8b-83bd-9d474fa97627 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e81b12a867af4d088a67a001e58ac9f9'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:00.915490 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-9ceb5a29-0b83-4a8b-83bd-9d474fa97627 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e81b12a867af4d088a67a001e58ac9f9'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:00.926005 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 976/1985] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:41:00 2022] DELETE /identity/v3/users/e81b12a867af4d088a67a001e58ac9f9 => generated 0 bytes in 489 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:00.928634 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8053d940-bec5-4ebb-91a7-82befade29da None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:00.934085 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8053d940-bec5-4ebb-91a7-82befade29da None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:00.934085 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8053d940-bec5-4ebb-91a7-82befade29da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:00.953218 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-8053d940-bec5-4ebb-91a7-82befade29da None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:00.954511 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-8053d940-bec5-4ebb-91a7-82befade29da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:01.033669 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 977/1986] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:41:01.047102 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-1108031f-76de-4707-adb4-c52165dc5edb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '33a7a8fa558b472da92a52cdd63bdbad', 'request_id': 'req-1108031f-76de-4707-adb4-c52165dc5edb'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:01.051790 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8053d940-bec5-4ebb-91a7-82befade29da None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:01.056196 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-8053d940-bec5-4ebb-91a7-82befade29da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:01.056344 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-1108031f-76de-4707-adb4-c52165dc5edb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '33a7a8fa558b472da92a52cdd63bdbad', 'request_id': 'req-1108031f-76de-4707-adb4-c52165dc5edb'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:01.136507 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8053d940-bec5-4ebb-91a7-82befade29da None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:01.139524 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-1108031f-76de-4707-adb4-c52165dc5edb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '33a7a8fa558b472da92a52cdd63bdbad', 'request_id': 'req-1108031f-76de-4707-adb4-c52165dc5edb'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:01.154792 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-1108031f-76de-4707-adb4-c52165dc5edb admin admin] The token cache is being invalidate because project 33a7a8fa558b472da92a52cdd63bdbad was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:41:01.155255 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-1108031f-76de-4707-adb4-c52165dc5edb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:01.157262 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-1108031f-76de-4707-adb4-c52165dc5edb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:01.159717 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1010/1987] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:41:00 2022] DELETE /identity/v3/projects/33a7a8fa558b472da92a52cdd63bdbad => generated 0 bytes in 330 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:41:01.166077 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9df1123-875a-4246-a876-90303baffeea None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:01.166786 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9df1123-875a-4246-a876-90303baffeea None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:01.167334 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9df1123-875a-4246-a876-90303baffeea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:01.190620 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-d9df1123-875a-4246-a876-90303baffeea None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:01.192748 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-d9df1123-875a-4246-a876-90303baffeea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:01.229795 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9219374-225d-4e41-a415-5dd4992bc820 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:01.230865 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9219374-225d-4e41-a415-5dd4992bc820 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:01.236241 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9219374-225d-4e41-a415-5dd4992bc820 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:01.236241 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9219374-225d-4e41-a415-5dd4992bc820 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:01.236241 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-b9219374-225d-4e41-a415-5dd4992bc820 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:01.284607 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1011/1988] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:01.293608 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72918acc-9e06-4835-9b4e-4cd2f1e0c156 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:01.293909 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72918acc-9e06-4835-9b4e-4cd2f1e0c156 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:01.294197 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72918acc-9e06-4835-9b4e-4cd2f1e0c156 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:01.320410 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9219374-225d-4e41-a415-5dd4992bc820 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:01.322653 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9219374-225d-4e41-a415-5dd4992bc820 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:01.324893 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-72918acc-9e06-4835-9b4e-4cd2f1e0c156 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:01.326115 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-72918acc-9e06-4835-9b4e-4cd2f1e0c156 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:01.339721 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 978/1989] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:41:01 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 285 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:41:01.340388 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9219374-225d-4e41-a415-5dd4992bc820 service neutron] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:01.342656 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-b9219374-225d-4e41-a415-5dd4992bc820 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:01.346596 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1012/1990] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:41:01.347274 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72918acc-9e06-4835-9b4e-4cd2f1e0c156 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:01.362455 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72918acc-9e06-4835-9b4e-4cd2f1e0c156 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:01.368400 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9219374-225d-4e41-a415-5dd4992bc820 service neutron] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:01.369875 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1c9e6a-67c1-4e07-9b73-d45c1a16b24b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:01.370773 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1c9e6a-67c1-4e07-9b73-d45c1a16b24b admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:01.371066 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1c9e6a-67c1-4e07-9b73-d45c1a16b24b admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:01.371344 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1c9e6a-67c1-4e07-9b73-d45c1a16b24b admin admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:01.372341 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f1c9e6a-67c1-4e07-9b73-d45c1a16b24b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:41:01.372650 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f1c9e6a-67c1-4e07-9b73-d45c1a16b24b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:01.374965 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f1c9e6a-67c1-4e07-9b73-d45c1a16b24b admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:01.379196 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a259a3a-5ab4-4eb3-9a93-c4313cdf5ff6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:01.382131 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a259a3a-5ab4-4eb3-9a93-c4313cdf5ff6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:01.382131 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a259a3a-5ab4-4eb3-9a93-c4313cdf5ff6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:01.382131 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a259a3a-5ab4-4eb3-9a93-c4313cdf5ff6 admin admin] PATH_INFO: `/v3/projects/a224f7b691294e7f89e6b5d8ee401624` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:01.382776 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a259a3a-5ab4-4eb3-9a93-c4313cdf5ff6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a224f7b691294e7f89e6b5d8ee401624)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:01.384972 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a259a3a-5ab4-4eb3-9a93-c4313cdf5ff6 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:01.400840 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1013/1991] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:41:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:41:01.404996 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1c9e6a-67c1-4e07-9b73-d45c1a16b24b admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:01.417118 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1c9e6a-67c1-4e07-9b73-d45c1a16b24b admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:01.422646 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a71b2ab-a9f1-446c-af7c-6076be1c1eb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:01.423455 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a71b2ab-a9f1-446c-af7c-6076be1c1eb2 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:01.423822 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a71b2ab-a9f1-446c-af7c-6076be1c1eb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:01.424325 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a71b2ab-a9f1-446c-af7c-6076be1c1eb2 admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:01.425104 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a71b2ab-a9f1-446c-af7c-6076be1c1eb2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:01.427135 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a71b2ab-a9f1-446c-af7c-6076be1c1eb2 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:01.448599 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:01.559247 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-8a259a3a-5ab4-4eb3-9a93-c4313cdf5ff6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a224f7b691294e7f89e6b5d8ee401624', 'request_id': 'req-8a259a3a-5ab4-4eb3-9a93-c4313cdf5ff6'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:01.568284 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-8a259a3a-5ab4-4eb3-9a93-c4313cdf5ff6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a224f7b691294e7f89e6b5d8ee401624', 'request_id': 'req-8a259a3a-5ab4-4eb3-9a93-c4313cdf5ff6'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:01.582926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-8a259a3a-5ab4-4eb3-9a93-c4313cdf5ff6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a224f7b691294e7f89e6b5d8ee401624', 'request_id': 'req-8a259a3a-5ab4-4eb3-9a93-c4313cdf5ff6'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:01.588308 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1014/1992] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:41:01 2022] POST /identity/v3/projects => generated 381 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:41:01.592537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a71b2ab-a9f1-446c-af7c-6076be1c1eb2 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:01.668660 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-8a259a3a-5ab4-4eb3-9a93-c4313cdf5ff6 admin admin] The token cache is being invalidate because project a224f7b691294e7f89e6b5d8ee401624 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:41:01.668660 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-8a259a3a-5ab4-4eb3-9a93-c4313cdf5ff6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:01.669131 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-8a259a3a-5ab4-4eb3-9a93-c4313cdf5ff6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:01.672303 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 979/1993] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:41:01 2022] DELETE /identity/v3/projects/a224f7b691294e7f89e6b5d8ee401624 => generated 0 bytes in 332 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:01.676362 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a259a3a-5ab4-4eb3-9a93-c4313cdf5ff6 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:01.678835 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-8a259a3a-5ab4-4eb3-9a93-c4313cdf5ff6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:01.716736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a71b2ab-a9f1-446c-af7c-6076be1c1eb2 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:01.723827 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbf4c81-37ac-4eb2-ac5b-3b06cb848114 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:01.724526 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbf4c81-37ac-4eb2-ac5b-3b06cb848114 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:01.725021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbf4c81-37ac-4eb2-ac5b-3b06cb848114 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:01.725302 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbf4c81-37ac-4eb2-ac5b-3b06cb848114 admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:01.726033 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edbf4c81-37ac-4eb2-ac5b-3b06cb848114 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:01.728802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edbf4c81-37ac-4eb2-ac5b-3b06cb848114 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:01.815330 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a259a3a-5ab4-4eb3-9a93-c4313cdf5ff6 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:01.823113 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c1716d-5e6c-49b6-896b-cdd8c6278790 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:01.824679 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c1716d-5e6c-49b6-896b-cdd8c6278790 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:01.828053 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c1716d-5e6c-49b6-896b-cdd8c6278790 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:01.828053 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c1716d-5e6c-49b6-896b-cdd8c6278790 admin admin] PATH_INFO: `/v3/users/d3bd428cb2a34f9683c7d7acb5195f36` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:01.828800 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c1716d-5e6c-49b6-896b-cdd8c6278790 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d3bd428cb2a34f9683c7d7acb5195f36)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:01.832442 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c1716d-5e6c-49b6-896b-cdd8c6278790 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:01.850478 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:01.862799 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1015/1994] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:41:01 2022] POST /identity/v3/users => generated 291 bytes in 268 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:41:01.863876 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbf4c81-37ac-4eb2-ac5b-3b06cb848114 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:01.865883 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-edbf4c81-37ac-4eb2-ac5b-3b06cb848114 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:01.880778 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbf4c81-37ac-4eb2-ac5b-3b06cb848114 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:01.885816 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148b22e1-6020-4a64-99bf-34359dda9b6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:01.886521 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148b22e1-6020-4a64-99bf-34359dda9b6b admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:01.886834 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148b22e1-6020-4a64-99bf-34359dda9b6b admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:01.887172 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148b22e1-6020-4a64-99bf-34359dda9b6b admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:01.887919 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148b22e1-6020-4a64-99bf-34359dda9b6b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:01.889520 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148b22e1-6020-4a64-99bf-34359dda9b6b admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:01.901041 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1016/1995] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:01 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:41:01.904890 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148b22e1-6020-4a64-99bf-34359dda9b6b admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:01.916434 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148b22e1-6020-4a64-99bf-34359dda9b6b admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:01.921329 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f016f0c-1fb8-405e-9648-08dc6ca5d15e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:01.922045 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f016f0c-1fb8-405e-9648-08dc6ca5d15e admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:01.922386 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f016f0c-1fb8-405e-9648-08dc6ca5d15e admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:01.922700 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f016f0c-1fb8-405e-9648-08dc6ca5d15e admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:01.923446 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f016f0c-1fb8-405e-9648-08dc6ca5d15e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:01.925059 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f016f0c-1fb8-405e-9648-08dc6ca5d15e admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:01.936444 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1017/1996] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:01 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 12 17:41:01.940813 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f016f0c-1fb8-405e-9648-08dc6ca5d15e admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:01.950845 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f016f0c-1fb8-405e-9648-08dc6ca5d15e admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:01.956261 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba24e399-88da-426a-9cbd-4f7e605461a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:01.957181 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba24e399-88da-426a-9cbd-4f7e605461a9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:01.957472 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba24e399-88da-426a-9cbd-4f7e605461a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:01.957756 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba24e399-88da-426a-9cbd-4f7e605461a9 admin admin] PATH_INFO: `/v3/projects/7a4e997384574368b02ae1c38be768e8/users/c3d16229c134464d8326bdd874f7c180/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:01.976990 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba24e399-88da-426a-9cbd-4f7e605461a9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7a4e997384574368b02ae1c38be768e8, user_id=c3d16229c134464d8326bdd874f7c180, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:01.980062 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba24e399-88da-426a-9cbd-4f7e605461a9 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:02.085316 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-68c1716d-5e6c-49b6-896b-cdd8c6278790 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd3bd428cb2a34f9683c7d7acb5195f36'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:02.145822 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1018/1997] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:41:01 2022] PUT /identity/v3/projects/7a4e997384574368b02ae1c38be768e8/users/c3d16229c134464d8326bdd874f7c180/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:41:02.149336 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba24e399-88da-426a-9cbd-4f7e605461a9 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:02.158153 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-ba24e399-88da-426a-9cbd-4f7e605461a9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:02.225694 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-68c1716d-5e6c-49b6-896b-cdd8c6278790 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd3bd428cb2a34f9683c7d7acb5195f36'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:02.227010 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-68c1716d-5e6c-49b6-896b-cdd8c6278790 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd3bd428cb2a34f9683c7d7acb5195f36'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:02.242998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-68c1716d-5e6c-49b6-896b-cdd8c6278790 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd3bd428cb2a34f9683c7d7acb5195f36'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:02.243153 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba24e399-88da-426a-9cbd-4f7e605461a9 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:02.246224 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc007554-96e5-44a6-8d60-802f031ef8f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:02.247221 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc007554-96e5-44a6-8d60-802f031ef8f7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:02.247562 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc007554-96e5-44a6-8d60-802f031ef8f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:02.247914 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc007554-96e5-44a6-8d60-802f031ef8f7 admin admin] PATH_INFO: `/v3/projects/c8c850e77a524184a741b0857200d0a5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:02.249197 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc007554-96e5-44a6-8d60-802f031ef8f7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c8c850e77a524184a741b0857200d0a5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:02.251131 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-68c1716d-5e6c-49b6-896b-cdd8c6278790 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd3bd428cb2a34f9683c7d7acb5195f36'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:02.252198 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc007554-96e5-44a6-8d60-802f031ef8f7 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:02.279644 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 980/1998] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:41:01 2022] DELETE /identity/v3/users/d3bd428cb2a34f9683c7d7acb5195f36 => generated 0 bytes in 604 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:02.282926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8707de8-6cc4-4e11-bc70-49428f78236c None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:02.282926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8707de8-6cc4-4e11-bc70-49428f78236c None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:02.282926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8707de8-6cc4-4e11-bc70-49428f78236c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:02.317113 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-a8707de8-6cc4-4e11-bc70-49428f78236c None None] MFA Rules not processed for user `c3d16229c134464d8326bdd874f7c180`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:02.318877 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-a8707de8-6cc4-4e11-bc70-49428f78236c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:02.433871 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 981/1999] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:02 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:02.437465 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-bc007554-96e5-44a6-8d60-802f031ef8f7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c8c850e77a524184a741b0857200d0a5', 'request_id': 'req-bc007554-96e5-44a6-8d60-802f031ef8f7'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:02.440537 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8707de8-6cc4-4e11-bc70-49428f78236c None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:02.444434 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-a8707de8-6cc4-4e11-bc70-49428f78236c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:02.446491 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-bc007554-96e5-44a6-8d60-802f031ef8f7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c8c850e77a524184a741b0857200d0a5', 'request_id': 'req-bc007554-96e5-44a6-8d60-802f031ef8f7'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:02.526038 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8707de8-6cc4-4e11-bc70-49428f78236c None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:02.533939 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-bc007554-96e5-44a6-8d60-802f031ef8f7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c8c850e77a524184a741b0857200d0a5', 'request_id': 'req-bc007554-96e5-44a6-8d60-802f031ef8f7'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:02.548048 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-bc007554-96e5-44a6-8d60-802f031ef8f7 admin admin] The token cache is being invalidate because project c8c850e77a524184a741b0857200d0a5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:41:02.548364 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-bc007554-96e5-44a6-8d60-802f031ef8f7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:02.549172 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-bc007554-96e5-44a6-8d60-802f031ef8f7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:02.551052 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1019/2000] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:41:02 2022] DELETE /identity/v3/projects/c8c850e77a524184a741b0857200d0a5 => generated 0 bytes in 404 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:02.558456 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7f3c75-0785-4d3e-8abf-23dc4891110b None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:02.558808 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7f3c75-0785-4d3e-8abf-23dc4891110b None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:02.559250 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7f3c75-0785-4d3e-8abf-23dc4891110b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:02.598462 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-2d7f3c75-0785-4d3e-8abf-23dc4891110b None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:02.600058 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-2d7f3c75-0785-4d3e-8abf-23dc4891110b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:02.610905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848449c8-245a-4462-b7a1-9a230e34ea0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:02.611591 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848449c8-245a-4462-b7a1-9a230e34ea0a admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:02.611883 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848449c8-245a-4462-b7a1-9a230e34ea0a admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:02.612234 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848449c8-245a-4462-b7a1-9a230e34ea0a admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:02.612923 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-848449c8-245a-4462-b7a1-9a230e34ea0a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:02.614914 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-848449c8-245a-4462-b7a1-9a230e34ea0a admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:02.670121 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1020/2001] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:41:02.680350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc4f3b5-8b3d-4a22-be81-6061c61b3d21 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:02.680350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc4f3b5-8b3d-4a22-be81-6061c61b3d21 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:02.680350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc4f3b5-8b3d-4a22-be81-6061c61b3d21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:02.699358 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-efc4f3b5-8b3d-4a22-be81-6061c61b3d21 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:02.700601 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-efc4f3b5-8b3d-4a22-be81-6061c61b3d21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:02.709153 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1021/2002] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:41:02.714159 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 982/2003] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:41:02 2022] POST /identity/v3/projects => generated 381 bytes in 274 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:41:02.719209 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc4f3b5-8b3d-4a22-be81-6061c61b3d21 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:02.720895 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848449c8-245a-4462-b7a1-9a230e34ea0a admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:02.722650 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-848449c8-245a-4462-b7a1-9a230e34ea0a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:02.729430 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc4f3b5-8b3d-4a22-be81-6061c61b3d21 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:02.799944 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e0de2d-2562-4f00-97f5-4420777ca5ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:02.804250 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e0de2d-2562-4f00-97f5-4420777ca5ef admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:02.804250 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e0de2d-2562-4f00-97f5-4420777ca5ef admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:02.804948 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e0de2d-2562-4f00-97f5-4420777ca5ef admin admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:02.806626 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e0de2d-2562-4f00-97f5-4420777ca5ef admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:41:02.806941 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e0de2d-2562-4f00-97f5-4420777ca5ef admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:02.809999 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e0de2d-2562-4f00-97f5-4420777ca5ef admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:02.822021 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848449c8-245a-4462-b7a1-9a230e34ea0a admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:02.828934 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19bb2158-be2f-4ff3-8ef3-65dd4414c4c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:02.830024 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19bb2158-be2f-4ff3-8ef3-65dd4414c4c9 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:02.830531 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19bb2158-be2f-4ff3-8ef3-65dd4414c4c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:02.830931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19bb2158-be2f-4ff3-8ef3-65dd4414c4c9 admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:02.831786 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19bb2158-be2f-4ff3-8ef3-65dd4414c4c9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:02.834461 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19bb2158-be2f-4ff3-8ef3-65dd4414c4c9 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:02.838582 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1022/2004] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:41:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 120 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:41:02.840468 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e0de2d-2562-4f00-97f5-4420777ca5ef admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:02.844526 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-63e0de2d-2562-4f00-97f5-4420777ca5ef admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:02.869301 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e0de2d-2562-4f00-97f5-4420777ca5ef admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:02.875918 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1821fd8b-8b19-46e4-b1f7-9055f27f24a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:02.876761 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1821fd8b-8b19-46e4-b1f7-9055f27f24a1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:02.877231 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1821fd8b-8b19-46e4-b1f7-9055f27f24a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:02.877519 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1821fd8b-8b19-46e4-b1f7-9055f27f24a1 admin admin] PATH_INFO: `/v3/projects/fb0146591fe543e383d4c5aa20109336` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:02.878937 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1821fd8b-8b19-46e4-b1f7-9055f27f24a1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fb0146591fe543e383d4c5aa20109336)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:02.881002 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1821fd8b-8b19-46e4-b1f7-9055f27f24a1 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:02.957160 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 983/2005] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:41:02 2022] POST /identity/v3/users => generated 297 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:41:02.957160 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19bb2158-be2f-4ff3-8ef3-65dd4414c4c9 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:02.965330 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19bb2158-be2f-4ff3-8ef3-65dd4414c4c9 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:02.970728 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1710bf35-6110-4f36-9d1b-dcb7f8f43bc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:02.971444 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1710bf35-6110-4f36-9d1b-dcb7f8f43bc8 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:02.971743 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1710bf35-6110-4f36-9d1b-dcb7f8f43bc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:02.972029 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1710bf35-6110-4f36-9d1b-dcb7f8f43bc8 admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:02.972752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1710bf35-6110-4f36-9d1b-dcb7f8f43bc8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:02.974736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1710bf35-6110-4f36-9d1b-dcb7f8f43bc8 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:03.018904 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-1821fd8b-8b19-46e4-b1f7-9055f27f24a1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fb0146591fe543e383d4c5aa20109336', 'request_id': 'req-1821fd8b-8b19-46e4-b1f7-9055f27f24a1'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:03.027670 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-1821fd8b-8b19-46e4-b1f7-9055f27f24a1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fb0146591fe543e383d4c5aa20109336', 'request_id': 'req-1821fd8b-8b19-46e4-b1f7-9055f27f24a1'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:03.090501 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 984/2006] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:41:02 2022] POST /identity/v3/projects => generated 379 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:41:03.090501 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1710bf35-6110-4f36-9d1b-dcb7f8f43bc8 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:03.107094 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-1821fd8b-8b19-46e4-b1f7-9055f27f24a1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fb0146591fe543e383d4c5aa20109336', 'request_id': 'req-1821fd8b-8b19-46e4-b1f7-9055f27f24a1'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:03.144139 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-1821fd8b-8b19-46e4-b1f7-9055f27f24a1 admin admin] The token cache is being invalidate because project fb0146591fe543e383d4c5aa20109336 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:41:03.144597 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-1821fd8b-8b19-46e4-b1f7-9055f27f24a1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:03.145269 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-1821fd8b-8b19-46e4-b1f7-9055f27f24a1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:03.147094 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1023/2007] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:41:02 2022] DELETE /identity/v3/projects/fb0146591fe543e383d4c5aa20109336 => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:03.147892 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1821fd8b-8b19-46e4-b1f7-9055f27f24a1 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:03.149425 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-1821fd8b-8b19-46e4-b1f7-9055f27f24a1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:03.191458 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1710bf35-6110-4f36-9d1b-dcb7f8f43bc8 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:03.199489 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecfc2c8-bb41-45c1-a985-f5033f0f71fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:03.199489 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecfc2c8-bb41-45c1-a985-f5033f0f71fb admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:03.199489 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecfc2c8-bb41-45c1-a985-f5033f0f71fb admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:03.199489 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecfc2c8-bb41-45c1-a985-f5033f0f71fb admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:03.200204 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecfc2c8-bb41-45c1-a985-f5033f0f71fb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:03.200363 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecfc2c8-bb41-45c1-a985-f5033f0f71fb admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:03.215510 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 985/2008] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:03 2022] GET /identity/v3/roles => generated 1420 bytes in 128 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:41:03.217774 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cffac34-9e25-4add-9ef2-9fd55d13889c None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:03.218409 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cffac34-9e25-4add-9ef2-9fd55d13889c None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:03.223149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:03.223742 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cffac34-9e25-4add-9ef2-9fd55d13889c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:03.247717 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-2cffac34-9e25-4add-9ef2-9fd55d13889c None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:03.248956 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-2cffac34-9e25-4add-9ef2-9fd55d13889c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:03.252275 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1821fd8b-8b19-46e4-b1f7-9055f27f24a1 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:03.257299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a51df3c-5ca8-43a4-91d3-9a3ff46d121b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:03.257926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 986/2009] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:03.258092 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a51df3c-5ca8-43a4-91d3-9a3ff46d121b admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:03.258382 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a51df3c-5ca8-43a4-91d3-9a3ff46d121b admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:03.258742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a51df3c-5ca8-43a4-91d3-9a3ff46d121b admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:03.258919 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cffac34-9e25-4add-9ef2-9fd55d13889c None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:03.259420 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a51df3c-5ca8-43a4-91d3-9a3ff46d121b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:03.261467 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a51df3c-5ca8-43a4-91d3-9a3ff46d121b admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:03.265952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-2cffac34-9e25-4add-9ef2-9fd55d13889c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:03.273137 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:03.278403 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cffac34-9e25-4add-9ef2-9fd55d13889c None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:03.283409 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb905fd-fa64-4da5-acfe-5649e3e06492 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:03.284332 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb905fd-fa64-4da5-acfe-5649e3e06492 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:03.284616 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb905fd-fa64-4da5-acfe-5649e3e06492 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:03.284900 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfb905fd-fa64-4da5-acfe-5649e3e06492 admin admin] PATH_INFO: `/v3/projects/9e8ead470193456d8d56a23791099482/users/ef296a8ff467471abc46b6f80f16287c/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:03.305493 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb905fd-fa64-4da5-acfe-5649e3e06492 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9e8ead470193456d8d56a23791099482, user_id=ef296a8ff467471abc46b6f80f16287c, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:03.309691 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfb905fd-fa64-4da5-acfe-5649e3e06492 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:03.361922 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1024/2010] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:41:03 2022] POST /identity/v3/users => generated 290 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:41:03.361922 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63722c00-8281-48bf-aa6e-a44d41409909 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:03.361922 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63722c00-8281-48bf-aa6e-a44d41409909 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:03.361922 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63722c00-8281-48bf-aa6e-a44d41409909 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:03.384869 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-63722c00-8281-48bf-aa6e-a44d41409909 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:03.386014 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-63722c00-8281-48bf-aa6e-a44d41409909 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:03.395223 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1025/2011] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:03.403235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63722c00-8281-48bf-aa6e-a44d41409909 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:03.412876 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63722c00-8281-48bf-aa6e-a44d41409909 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:03.421918 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9a8961-6380-4b99-8652-71939f40a165 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:03.422753 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9a8961-6380-4b99-8652-71939f40a165 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:03.423146 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 987/2012] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:41:03 2022] PUT /identity/v3/projects/9e8ead470193456d8d56a23791099482/users/ef296a8ff467471abc46b6f80f16287c/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:03.424047 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9a8961-6380-4b99-8652-71939f40a165 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:03.424733 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9a8961-6380-4b99-8652-71939f40a165 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:03.425317 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb905fd-fa64-4da5-acfe-5649e3e06492 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:03.426214 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c9a8961-6380-4b99-8652-71939f40a165 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:03.428794 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c9a8961-6380-4b99-8652-71939f40a165 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:03.445519 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1026/2013] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:03 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:41:03.449417 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9a8961-6380-4b99-8652-71939f40a165 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:03.461341 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9a8961-6380-4b99-8652-71939f40a165 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:03.543121 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfb905fd-fa64-4da5-acfe-5649e3e06492 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:03.551725 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29dceb53-75a2-4d8e-8b5c-bb41b9ac2cdf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:03.552901 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29dceb53-75a2-4d8e-8b5c-bb41b9ac2cdf admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:03.553326 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29dceb53-75a2-4d8e-8b5c-bb41b9ac2cdf admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:03.554059 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29dceb53-75a2-4d8e-8b5c-bb41b9ac2cdf admin admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:03.554867 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29dceb53-75a2-4d8e-8b5c-bb41b9ac2cdf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:41:03.555435 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29dceb53-75a2-4d8e-8b5c-bb41b9ac2cdf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:03.558786 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29dceb53-75a2-4d8e-8b5c-bb41b9ac2cdf admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:03.580807 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa33b6b-f71b-43aa-acd0-29750fdb46ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:03.582265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa33b6b-f71b-43aa-acd0-29750fdb46ff admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:03.583193 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa33b6b-f71b-43aa-acd0-29750fdb46ff admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:03.583708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa33b6b-f71b-43aa-acd0-29750fdb46ff admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:03.584730 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 988/2014] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:41:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:41:03.584933 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa33b6b-f71b-43aa-acd0-29750fdb46ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:03.586280 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa33b6b-f71b-43aa-acd0-29750fdb46ff admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:03.592081 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29dceb53-75a2-4d8e-8b5c-bb41b9ac2cdf admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:03.603534 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29dceb53-75a2-4d8e-8b5c-bb41b9ac2cdf admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:03.608566 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4c4582-03e5-43c3-9d91-aece9881c920 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:03.613701 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1027/2015] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:03 2022] GET /identity/v3/roles => generated 1420 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:41:03.613881 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4c4582-03e5-43c3-9d91-aece9881c920 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:03.613881 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4c4582-03e5-43c3-9d91-aece9881c920 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:03.613881 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4c4582-03e5-43c3-9d91-aece9881c920 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:03.614123 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa33b6b-f71b-43aa-acd0-29750fdb46ff admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:03.614210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d4c4582-03e5-43c3-9d91-aece9881c920 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:03.614401 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d4c4582-03e5-43c3-9d91-aece9881c920 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:03.632416 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa33b6b-f71b-43aa-acd0-29750fdb46ff admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:03.638529 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 989/2016] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:03 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:41:03.638529 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4c4582-03e5-43c3-9d91-aece9881c920 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:03.642507 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024b75f6-dae9-4f58-b9be-df732e33d4cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:03.643204 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024b75f6-dae9-4f58-b9be-df732e33d4cd admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:03.643490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024b75f6-dae9-4f58-b9be-df732e33d4cd admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:03.643774 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024b75f6-dae9-4f58-b9be-df732e33d4cd admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:03.644519 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-024b75f6-dae9-4f58-b9be-df732e33d4cd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:03.646764 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-024b75f6-dae9-4f58-b9be-df732e33d4cd admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:03.657651 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4c4582-03e5-43c3-9d91-aece9881c920 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:03.664273 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe1a47e-7973-4501-aa7f-f967c1f4dde1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:03.666594 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe1a47e-7973-4501-aa7f-f967c1f4dde1 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:03.666594 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe1a47e-7973-4501-aa7f-f967c1f4dde1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:03.666594 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe1a47e-7973-4501-aa7f-f967c1f4dde1 admin admin] PATH_INFO: `/v3/projects/9e8ead470193456d8d56a23791099482/users/ef296a8ff467471abc46b6f80f16287c/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:03.668285 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe1a47e-7973-4501-aa7f-f967c1f4dde1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9e8ead470193456d8d56a23791099482, user_id=ef296a8ff467471abc46b6f80f16287c, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:03.671326 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abe1a47e-7973-4501-aa7f-f967c1f4dde1 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:03.684350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:03.686569 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 990/2017] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:41:03 2022] PUT /identity/v3/projects/9e8ead470193456d8d56a23791099482/users/ef296a8ff467471abc46b6f80f16287c/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:03.686569 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe1a47e-7973-4501-aa7f-f967c1f4dde1 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:03.730873 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1028/2018] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:41:03 2022] POST /identity/v3/projects => generated 371 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:41:03.736124 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b695fb97-46b2-4b92-9d56-31b58cfd8a45 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:03.736424 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b695fb97-46b2-4b92-9d56-31b58cfd8a45 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:03.736699 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b695fb97-46b2-4b92-9d56-31b58cfd8a45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:03.774767 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe1a47e-7973-4501-aa7f-f967c1f4dde1 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:03.786643 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-b695fb97-46b2-4b92-9d56-31b58cfd8a45 None None] MFA Rules not processed for user `ef296a8ff467471abc46b6f80f16287c`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:03.790997 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-b695fb97-46b2-4b92-9d56-31b58cfd8a45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:03.791661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:03.874889 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286f8e02-65d8-423e-922a-e4a0650126b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:03.877247 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286f8e02-65d8-423e-922a-e4a0650126b9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:03.877758 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286f8e02-65d8-423e-922a-e4a0650126b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:03.878289 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286f8e02-65d8-423e-922a-e4a0650126b9 admin admin] PATH_INFO: `/v3/projects/e4b2b488b746454b9f93d67fa3c97809/users/bde6ee9df10146d98991b0d4bb37fc48/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:03.903232 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1029/2019] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:03 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:03.904301 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b695fb97-46b2-4b92-9d56-31b58cfd8a45 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:03.911577 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286f8e02-65d8-423e-922a-e4a0650126b9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e4b2b488b746454b9f93d67fa3c97809, user_id=bde6ee9df10146d98991b0d4bb37fc48, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:03.917286 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286f8e02-65d8-423e-922a-e4a0650126b9 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:03.923639 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b695fb97-46b2-4b92-9d56-31b58cfd8a45 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:03.929113 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1748cf1c-dc8e-4e52-9dc0-53d983a64843 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:03.929859 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1748cf1c-dc8e-4e52-9dc0-53d983a64843 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:03.930155 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1748cf1c-dc8e-4e52-9dc0-53d983a64843 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:03.930468 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1748cf1c-dc8e-4e52-9dc0-53d983a64843 admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:03.931209 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1748cf1c-dc8e-4e52-9dc0-53d983a64843 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:03.933334 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1748cf1c-dc8e-4e52-9dc0-53d983a64843 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:03.998222 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 991/2020] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:41:03 2022] PUT /identity/v3/projects/e4b2b488b746454b9f93d67fa3c97809/users/bde6ee9df10146d98991b0d4bb37fc48/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 313 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:04.000066 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286f8e02-65d8-423e-922a-e4a0650126b9 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:04.004014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-286f8e02-65d8-423e-922a-e4a0650126b9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:04.047994 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1030/2021] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:41:03 2022] POST /identity/v3/users => generated 286 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:41:04.049566 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0390f6-c5d0-45ec-84c0-4672c01b2821 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:04.049946 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0390f6-c5d0-45ec-84c0-4672c01b2821 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:04.050942 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0390f6-c5d0-45ec-84c0-4672c01b2821 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:04.071936 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286f8e02-65d8-423e-922a-e4a0650126b9 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:04.073712 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-ba0390f6-c5d0-45ec-84c0-4672c01b2821 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:04.075497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-ba0390f6-c5d0-45ec-84c0-4672c01b2821 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:04.077889 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d19f82a-564d-4544-9eaf-1bea2fbd412d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:04.078999 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d19f82a-564d-4544-9eaf-1bea2fbd412d service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:04.078999 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d19f82a-564d-4544-9eaf-1bea2fbd412d service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:04.079392 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d19f82a-564d-4544-9eaf-1bea2fbd412d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:04.142846 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:DELETE=6,keystone:SELECT=89,keystone:INSERT=5 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:04.157335 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1031/2022] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:04.158130 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba0390f6-c5d0-45ec-84c0-4672c01b2821 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:04.160416 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d19f82a-564d-4544-9eaf-1bea2fbd412d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:04.161974 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d19f82a-564d-4544-9eaf-1bea2fbd412d service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:04.170808 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba0390f6-c5d0-45ec-84c0-4672c01b2821 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:04.178047 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc6bc32-9c7c-40b4-9676-5c5295838a01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:04.179844 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc6bc32-9c7c-40b4-9676-5c5295838a01 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:04.180323 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc6bc32-9c7c-40b4-9676-5c5295838a01 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:04.180740 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc6bc32-9c7c-40b4-9676-5c5295838a01 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:04.181862 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dc6bc32-9c7c-40b4-9676-5c5295838a01 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:04.183947 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dc6bc32-9c7c-40b4-9676-5c5295838a01 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:04.196345 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 992/2023] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:41:03 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 197 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:41:04.198377 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d19f82a-564d-4544-9eaf-1bea2fbd412d service cinder] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:04.201573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-0d19f82a-564d-4544-9eaf-1bea2fbd412d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:04.206934 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1032/2024] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:04 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:41:04.210331 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc6bc32-9c7c-40b4-9676-5c5295838a01 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:04.215294 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:04.228201 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc6bc32-9c7c-40b4-9676-5c5295838a01 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:04.235064 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf57302-76ad-4d9b-b186-7c51d66911bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:04.236469 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf57302-76ad-4d9b-b186-7c51d66911bc admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:04.236839 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf57302-76ad-4d9b-b186-7c51d66911bc admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:04.237195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf57302-76ad-4d9b-b186-7c51d66911bc admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:04.238047 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf57302-76ad-4d9b-b186-7c51d66911bc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:04.239889 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf57302-76ad-4d9b-b186-7c51d66911bc admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:04.260586 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1033/2025] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:04 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:41:04.265066 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf57302-76ad-4d9b-b186-7c51d66911bc admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:04.286661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf57302-76ad-4d9b-b186-7c51d66911bc admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:04.292614 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9eaddb-1279-4568-8763-9c909f6180f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:04.293856 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9eaddb-1279-4568-8763-9c909f6180f8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:04.294318 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9eaddb-1279-4568-8763-9c909f6180f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:04.294853 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9eaddb-1279-4568-8763-9c909f6180f8 admin admin] PATH_INFO: `/v3/projects/f61730662ccb415ab0cf4b92a5866387/users/80a0cda11f8144099d464a037183408d/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:04.333224 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9eaddb-1279-4568-8763-9c909f6180f8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f61730662ccb415ab0cf4b92a5866387, user_id=80a0cda11f8144099d464a037183408d, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:04.336728 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9eaddb-1279-4568-8763-9c909f6180f8 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:04.338873 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d19f82a-564d-4544-9eaf-1bea2fbd412d service cinder] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:04.347948 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f408b1-989f-47a9-87d0-f3e3e969e98d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:04.356727 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f408b1-989f-47a9-87d0-f3e3e969e98d service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:04.356727 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f408b1-989f-47a9-87d0-f3e3e969e98d service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:04.356727 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f408b1-989f-47a9-87d0-f3e3e969e98d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:04.365780 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29f408b1-989f-47a9-87d0-f3e3e969e98d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:04.367506 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29f408b1-989f-47a9-87d0-f3e3e969e98d service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:04.380549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 993/2026] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:41:04 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 183 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:41:04.439681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1034/2027] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:41:04 2022] PUT /identity/v3/projects/f61730662ccb415ab0cf4b92a5866387/users/80a0cda11f8144099d464a037183408d/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:04.450393 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59a84be-f328-4090-98ad-c9f4806611c8 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:04.450735 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59a84be-f328-4090-98ad-c9f4806611c8 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:04.451783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c59a84be-f328-4090-98ad-c9f4806611c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:04.482173 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-c59a84be-f328-4090-98ad-c9f4806611c8 None None] MFA Rules not processed for user `80a0cda11f8144099d464a037183408d`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:04.488802 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-c59a84be-f328-4090-98ad-c9f4806611c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:04.501335 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9eaddb-1279-4568-8763-9c909f6180f8 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:04.585116 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9eaddb-1279-4568-8763-9c909f6180f8 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:04.589033 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53bf038-e084-4481-a9ff-13886681f5be service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:04.589722 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53bf038-e084-4481-a9ff-13886681f5be service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:04.590342 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53bf038-e084-4481-a9ff-13886681f5be service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:04.590342 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b53bf038-e084-4481-a9ff-13886681f5be service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:04.592302 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-b53bf038-e084-4481-a9ff-13886681f5be service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:04.607709 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 994/2028] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:04 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:04.612888 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59a84be-f328-4090-98ad-c9f4806611c8 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:04.628607 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:04.726117 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c59a84be-f328-4090-98ad-c9f4806611c8 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:04.729996 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:04.739888 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7853db44-221a-434a-b402-51a8f11d7ea2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:04.740833 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7853db44-221a-434a-b402-51a8f11d7ea2 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:04.741441 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7853db44-221a-434a-b402-51a8f11d7ea2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:04.741863 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7853db44-221a-434a-b402-51a8f11d7ea2 admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:04.746787 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7853db44-221a-434a-b402-51a8f11d7ea2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:04.751975 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b53bf038-e084-4481-a9ff-13886681f5be service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:04.753806 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b53bf038-e084-4481-a9ff-13886681f5be service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:04.760540 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7853db44-221a-434a-b402-51a8f11d7ea2 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:04.770590 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1035/2029] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:41:04 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 269 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:41:04.902017 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 995/2030] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:41:04 2022] POST /identity/v3/projects => generated 373 bytes in 290 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:41:04.907241 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53bf038-e084-4481-a9ff-13886681f5be service cinder] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:04.919954 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b53bf038-e084-4481-a9ff-13886681f5be service cinder] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:05.009023 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:05.014903 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e335d73-dfcd-4538-913e-67d09e47f34f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:05.015774 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e335d73-dfcd-4538-913e-67d09e47f34f admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:05.016128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e335d73-dfcd-4538-913e-67d09e47f34f admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:05.018419 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e335d73-dfcd-4538-913e-67d09e47f34f admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:05.018684 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e335d73-dfcd-4538-913e-67d09e47f34f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:05.021436 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e335d73-dfcd-4538-913e-67d09e47f34f admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:05.157263 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1036/2031] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:41:04 2022] POST /identity/v3/users => generated 293 bytes in 248 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:41:05.159159 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7853db44-221a-434a-b402-51a8f11d7ea2 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:05.172636 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7853db44-221a-434a-b402-51a8f11d7ea2 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:05.179263 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19991bbc-6e2b-4cbb-aa69-ad0ca553370c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:05.183198 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19991bbc-6e2b-4cbb-aa69-ad0ca553370c admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:05.183198 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19991bbc-6e2b-4cbb-aa69-ad0ca553370c admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:05.183198 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19991bbc-6e2b-4cbb-aa69-ad0ca553370c admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:05.183198 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19991bbc-6e2b-4cbb-aa69-ad0ca553370c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:05.183580 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19991bbc-6e2b-4cbb-aa69-ad0ca553370c admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:05.197161 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 996/2032] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:05 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:41:05.202328 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e335d73-dfcd-4538-913e-67d09e47f34f admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:05.212978 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e335d73-dfcd-4538-913e-67d09e47f34f admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:05.217713 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebc2007-679d-4347-b7d0-71b678b14a6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:05.219384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebc2007-679d-4347-b7d0-71b678b14a6b admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:05.219822 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebc2007-679d-4347-b7d0-71b678b14a6b admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:05.220230 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebc2007-679d-4347-b7d0-71b678b14a6b admin admin] PATH_INFO: `/v3/projects/0cb0df69274e45cf99f06fe33df9bbac/users/6dd1a35ee7eb4b478ee70e60bc255134/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:05.252652 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ebc2007-679d-4347-b7d0-71b678b14a6b admin admin] RBAC: Authorizing `identity:create_grant(project_id=0cb0df69274e45cf99f06fe33df9bbac, user_id=6dd1a35ee7eb4b478ee70e60bc255134, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:05.258519 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ebc2007-679d-4347-b7d0-71b678b14a6b admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:05.404867 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1037/2033] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:41:05 2022] PUT /identity/v3/projects/0cb0df69274e45cf99f06fe33df9bbac/users/6dd1a35ee7eb4b478ee70e60bc255134/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:05.409599 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19991bbc-6e2b-4cbb-aa69-ad0ca553370c admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:05.426778 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19991bbc-6e2b-4cbb-aa69-ad0ca553370c admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:05.502845 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c647d15-23f0-4560-a4f0-e3b39f3747e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:05.503583 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c647d15-23f0-4560-a4f0-e3b39f3747e5 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:05.503928 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c647d15-23f0-4560-a4f0-e3b39f3747e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:05.504223 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c647d15-23f0-4560-a4f0-e3b39f3747e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:05.504969 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c647d15-23f0-4560-a4f0-e3b39f3747e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:05.506784 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c647d15-23f0-4560-a4f0-e3b39f3747e5 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:05.522468 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 997/2034] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:05 2022] GET /identity/v3/roles => generated 1420 bytes in 114 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:41:05.527066 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebc2007-679d-4347-b7d0-71b678b14a6b admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:05.546174 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebc2007-679d-4347-b7d0-71b678b14a6b admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:05.555893 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6894efb9-9886-4004-8c8a-154e612e6f7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:05.557458 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6894efb9-9886-4004-8c8a-154e612e6f7d admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:05.557975 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6894efb9-9886-4004-8c8a-154e612e6f7d admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:05.558559 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6894efb9-9886-4004-8c8a-154e612e6f7d admin admin] PATH_INFO: `/v3/projects/0cb0df69274e45cf99f06fe33df9bbac/users/6dd1a35ee7eb4b478ee70e60bc255134/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:05.563443 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6894efb9-9886-4004-8c8a-154e612e6f7d admin admin] RBAC: Authorizing `identity:create_grant(project_id=0cb0df69274e45cf99f06fe33df9bbac, user_id=6dd1a35ee7eb4b478ee70e60bc255134, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:05.570898 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6894efb9-9886-4004-8c8a-154e612e6f7d admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:05.593599 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1038/2035] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:41:05 2022] PUT /identity/v3/projects/0cb0df69274e45cf99f06fe33df9bbac/users/6dd1a35ee7eb4b478ee70e60bc255134/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:05.605477 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79913d46-89f6-4a9a-abd0-bd629d0577a8 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:05.606097 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79913d46-89f6-4a9a-abd0-bd629d0577a8 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:05.606402 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79913d46-89f6-4a9a-abd0-bd629d0577a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:05.640904 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-79913d46-89f6-4a9a-abd0-bd629d0577a8 None None] MFA Rules not processed for user `6dd1a35ee7eb4b478ee70e60bc255134`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:05.643178 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-79913d46-89f6-4a9a-abd0-bd629d0577a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:05.667297 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:05.741950 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 998/2036] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:05 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:05.752390 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6894efb9-9886-4004-8c8a-154e612e6f7d admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:05.754480 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-6894efb9-9886-4004-8c8a-154e612e6f7d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:05.805381 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:05.837429 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6894efb9-9886-4004-8c8a-154e612e6f7d admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:05.841797 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a7edec-6e43-4faf-bd55-44a7380469ad service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:05.842626 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a7edec-6e43-4faf-bd55-44a7380469ad service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:05.842947 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a7edec-6e43-4faf-bd55-44a7380469ad service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:05.844241 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a7edec-6e43-4faf-bd55-44a7380469ad service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:05.854948 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a7edec-6e43-4faf-bd55-44a7380469ad service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:05.856429 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a7edec-6e43-4faf-bd55-44a7380469ad service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:05.880069 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1039/2037] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:41:05 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:41:08.117992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79913d46-89f6-4a9a-abd0-bd629d0577a8 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:08.256439 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79913d46-89f6-4a9a-abd0-bd629d0577a8 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:08.261848 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb01885-7b2e-4ed9-bcc0-761cb5ad9fbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:08.262859 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb01885-7b2e-4ed9-bcc0-761cb5ad9fbc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:08.263235 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb01885-7b2e-4ed9-bcc0-761cb5ad9fbc admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:08.264169 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb01885-7b2e-4ed9-bcc0-761cb5ad9fbc admin admin] PATH_INFO: `/v3/users/80a0cda11f8144099d464a037183408d` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:08.265457 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcb01885-7b2e-4ed9-bcc0-761cb5ad9fbc admin admin] RBAC: Authorizing `identity:delete_user(user_id=80a0cda11f8144099d464a037183408d)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:08.275731 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcb01885-7b2e-4ed9-bcc0-761cb5ad9fbc admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:08.546872 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-fcb01885-7b2e-4ed9-bcc0-761cb5ad9fbc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '80a0cda11f8144099d464a037183408d'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:08.677139 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-fcb01885-7b2e-4ed9-bcc0-761cb5ad9fbc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '80a0cda11f8144099d464a037183408d'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:08.678038 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-fcb01885-7b2e-4ed9-bcc0-761cb5ad9fbc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '80a0cda11f8144099d464a037183408d'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:08.689991 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-fcb01885-7b2e-4ed9-bcc0-761cb5ad9fbc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '80a0cda11f8144099d464a037183408d'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:08.716588 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-fcb01885-7b2e-4ed9-bcc0-761cb5ad9fbc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '80a0cda11f8144099d464a037183408d'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:08.732841 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 999/2038] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:41:08 2022] DELETE /identity/v3/users/80a0cda11f8144099d464a037183408d => generated 0 bytes in 611 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:08.734789 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd6b133-7d5c-4bfa-a8f0-1bd8f6a88a34 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:08.735767 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd6b133-7d5c-4bfa-a8f0-1bd8f6a88a34 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:08.736308 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd6b133-7d5c-4bfa-a8f0-1bd8f6a88a34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:08.758676 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-9dd6b133-7d5c-4bfa-a8f0-1bd8f6a88a34 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:08.760920 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-9dd6b133-7d5c-4bfa-a8f0-1bd8f6a88a34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:08.856642 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1040/2039] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:41:08.869548 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb01885-7b2e-4ed9-bcc0-761cb5ad9fbc admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:08.871483 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-fcb01885-7b2e-4ed9-bcc0-761cb5ad9fbc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:08.952127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:09.007112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb01885-7b2e-4ed9-bcc0-761cb5ad9fbc admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:09.012007 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1a4dcd-5248-469f-8880-cdf7662abc15 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:09.013098 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1a4dcd-5248-469f-8880-cdf7662abc15 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:09.013651 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1a4dcd-5248-469f-8880-cdf7662abc15 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:09.014202 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f1a4dcd-5248-469f-8880-cdf7662abc15 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:09.032726 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f1a4dcd-5248-469f-8880-cdf7662abc15 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:09.034418 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f1a4dcd-5248-469f-8880-cdf7662abc15 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:09.049150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1000/2040] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:41:08 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 180 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:41:09.959887 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd6b133-7d5c-4bfa-a8f0-1bd8f6a88a34 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:09.961424 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-9dd6b133-7d5c-4bfa-a8f0-1bd8f6a88a34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:09.974115 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd6b133-7d5c-4bfa-a8f0-1bd8f6a88a34 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:09.981520 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7840f28d-b750-42b8-aca4-1dfaa641c2bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:09.982451 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7840f28d-b750-42b8-aca4-1dfaa641c2bd admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:09.982749 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7840f28d-b750-42b8-aca4-1dfaa641c2bd admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:09.983032 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7840f28d-b750-42b8-aca4-1dfaa641c2bd admin admin] PATH_INFO: `/v3/projects/f61730662ccb415ab0cf4b92a5866387` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:09.985834 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7840f28d-b750-42b8-aca4-1dfaa641c2bd admin admin] RBAC: Authorizing `identity:delete_project(project_id=f61730662ccb415ab0cf4b92a5866387)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:09.989824 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7840f28d-b750-42b8-aca4-1dfaa641c2bd admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:10.172983 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-7840f28d-b750-42b8-aca4-1dfaa641c2bd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f61730662ccb415ab0cf4b92a5866387', 'request_id': 'req-7840f28d-b750-42b8-aca4-1dfaa641c2bd'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:10.182734 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-7840f28d-b750-42b8-aca4-1dfaa641c2bd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f61730662ccb415ab0cf4b92a5866387', 'request_id': 'req-7840f28d-b750-42b8-aca4-1dfaa641c2bd'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:10.284266 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-7840f28d-b750-42b8-aca4-1dfaa641c2bd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f61730662ccb415ab0cf4b92a5866387', 'request_id': 'req-7840f28d-b750-42b8-aca4-1dfaa641c2bd'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:10.317396 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-7840f28d-b750-42b8-aca4-1dfaa641c2bd admin admin] The token cache is being invalidate because project f61730662ccb415ab0cf4b92a5866387 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:41:10.318037 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-7840f28d-b750-42b8-aca4-1dfaa641c2bd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:10.319722 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-7840f28d-b750-42b8-aca4-1dfaa641c2bd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:10.322378 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1041/2041] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:41:09 2022] DELETE /identity/v3/projects/f61730662ccb415ab0cf4b92a5866387 => generated 0 bytes in 363 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:10.326995 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1a4dcd-5248-469f-8880-cdf7662abc15 service neutron] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:10.342867 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f1a4dcd-5248-469f-8880-cdf7662abc15 service neutron] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:10.449872 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21377e14-aa2b-4e83-ae20-9b9a71cba624 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:10.451441 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21377e14-aa2b-4e83-ae20-9b9a71cba624 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:10.451965 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21377e14-aa2b-4e83-ae20-9b9a71cba624 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:10.452470 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21377e14-aa2b-4e83-ae20-9b9a71cba624 admin admin] PATH_INFO: `/v3/users/6dd1a35ee7eb4b478ee70e60bc255134` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:10.455489 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21377e14-aa2b-4e83-ae20-9b9a71cba624 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6dd1a35ee7eb4b478ee70e60bc255134)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:10.459324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21377e14-aa2b-4e83-ae20-9b9a71cba624 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:10.716192 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-21377e14-aa2b-4e83-ae20-9b9a71cba624 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6dd1a35ee7eb4b478ee70e60bc255134'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:10.797854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-21377e14-aa2b-4e83-ae20-9b9a71cba624 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6dd1a35ee7eb4b478ee70e60bc255134'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:10.800438 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-21377e14-aa2b-4e83-ae20-9b9a71cba624 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6dd1a35ee7eb4b478ee70e60bc255134'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:10.810139 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-21377e14-aa2b-4e83-ae20-9b9a71cba624 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6dd1a35ee7eb4b478ee70e60bc255134'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:10.835476 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-21377e14-aa2b-4e83-ae20-9b9a71cba624 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6dd1a35ee7eb4b478ee70e60bc255134'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:10.842453 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1001/2042] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:41:10 2022] DELETE /identity/v3/users/6dd1a35ee7eb4b478ee70e60bc255134 => generated 0 bytes in 516 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:11.737028 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7840f28d-b750-42b8-aca4-1dfaa641c2bd admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:11.739335 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-7840f28d-b750-42b8-aca4-1dfaa641c2bd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:11.769821 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7840f28d-b750-42b8-aca4-1dfaa641c2bd admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:11.867221 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:11.877891 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89e6f4f-80dd-49b1-b419-80c55afc91f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:11.878951 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89e6f4f-80dd-49b1-b419-80c55afc91f4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:11.879239 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89e6f4f-80dd-49b1-b419-80c55afc91f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:11.880183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89e6f4f-80dd-49b1-b419-80c55afc91f4 admin admin] PATH_INFO: `/v3/projects/0cb0df69274e45cf99f06fe33df9bbac` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:11.881786 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b89e6f4f-80dd-49b1-b419-80c55afc91f4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0cb0df69274e45cf99f06fe33df9bbac)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:11.883827 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b89e6f4f-80dd-49b1-b419-80c55afc91f4 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:12.027187 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-b89e6f4f-80dd-49b1-b419-80c55afc91f4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0cb0df69274e45cf99f06fe33df9bbac', 'request_id': 'req-b89e6f4f-80dd-49b1-b419-80c55afc91f4'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:12.034816 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-b89e6f4f-80dd-49b1-b419-80c55afc91f4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0cb0df69274e45cf99f06fe33df9bbac', 'request_id': 'req-b89e6f4f-80dd-49b1-b419-80c55afc91f4'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:12.163073 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-b89e6f4f-80dd-49b1-b419-80c55afc91f4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0cb0df69274e45cf99f06fe33df9bbac', 'request_id': 'req-b89e6f4f-80dd-49b1-b419-80c55afc91f4'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:12.181790 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-b89e6f4f-80dd-49b1-b419-80c55afc91f4 admin admin] The token cache is being invalidate because project 0cb0df69274e45cf99f06fe33df9bbac was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:41:12.182092 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-b89e6f4f-80dd-49b1-b419-80c55afc91f4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:12.182915 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-b89e6f4f-80dd-49b1-b419-80c55afc91f4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:12.185223 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1042/2043] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:41:11 2022] DELETE /identity/v3/projects/0cb0df69274e45cf99f06fe33df9bbac => generated 0 bytes in 449 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:12.191143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8eafc20-d0f5-422a-aedb-650d25ef4845 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:12.192141 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8eafc20-d0f5-422a-aedb-650d25ef4845 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:12.192632 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8eafc20-d0f5-422a-aedb-650d25ef4845 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:12.214773 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-b8eafc20-d0f5-422a-aedb-650d25ef4845 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:12.216304 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-b8eafc20-d0f5-422a-aedb-650d25ef4845 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:12.303801 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:12.332483 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1002/2044] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:12.345408 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf0cdd1-2beb-4bcd-8148-46200b303f1b None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:12.346324 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf0cdd1-2beb-4bcd-8148-46200b303f1b None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:12.346855 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf0cdd1-2beb-4bcd-8148-46200b303f1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:12.374840 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-6bf0cdd1-2beb-4bcd-8148-46200b303f1b None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:12.376103 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-6bf0cdd1-2beb-4bcd-8148-46200b303f1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:12.392282 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1043/2045] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:12.393225 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8eafc20-d0f5-422a-aedb-650d25ef4845 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:12.408346 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8eafc20-d0f5-422a-aedb-650d25ef4845 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:12.416343 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d99f42b-c781-43f5-98d3-db2c3df2b923 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:12.416343 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d99f42b-c781-43f5-98d3-db2c3df2b923 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:12.416343 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d99f42b-c781-43f5-98d3-db2c3df2b923 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:12.416343 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d99f42b-c781-43f5-98d3-db2c3df2b923 admin admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:12.416854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d99f42b-c781-43f5-98d3-db2c3df2b923 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:41:12.416854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d99f42b-c781-43f5-98d3-db2c3df2b923 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:12.417474 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d99f42b-c781-43f5-98d3-db2c3df2b923 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:12.431094 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1003/2046] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:41:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:41:12.436218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf0cdd1-2beb-4bcd-8148-46200b303f1b None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:12.446149 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf0cdd1-2beb-4bcd-8148-46200b303f1b None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:12.451146 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae77367e-8dec-4b6d-b8d7-ed86f2233531 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:12.451850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae77367e-8dec-4b6d-b8d7-ed86f2233531 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:12.452277 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae77367e-8dec-4b6d-b8d7-ed86f2233531 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:12.452563 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae77367e-8dec-4b6d-b8d7-ed86f2233531 admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:12.453763 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae77367e-8dec-4b6d-b8d7-ed86f2233531 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:12.456475 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae77367e-8dec-4b6d-b8d7-ed86f2233531 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:12.513073 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a129bdda-3edb-46ac-8777-afec3a26f85e None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:12.513579 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a129bdda-3edb-46ac-8777-afec3a26f85e None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:12.513872 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a129bdda-3edb-46ac-8777-afec3a26f85e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:12.564260 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-a129bdda-3edb-46ac-8777-afec3a26f85e None None] MFA Rules not processed for user `bde6ee9df10146d98991b0d4bb37fc48`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:12.569882 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-a129bdda-3edb-46ac-8777-afec3a26f85e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:12.588617 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1044/2047] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:41:12 2022] POST /identity/v3/projects => generated 393 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:41:12.592822 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae77367e-8dec-4b6d-b8d7-ed86f2233531 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:12.709472 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1004/2048] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:12 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:12.715687 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a129bdda-3edb-46ac-8777-afec3a26f85e None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:12.717378 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-a129bdda-3edb-46ac-8777-afec3a26f85e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:12.725938 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae77367e-8dec-4b6d-b8d7-ed86f2233531 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:12.734114 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b75bf9-54a7-4e16-bf45-3c2d19d5d081 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:12.734312 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a129bdda-3edb-46ac-8777-afec3a26f85e None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:12.734555 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b75bf9-54a7-4e16-bf45-3c2d19d5d081 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:12.734555 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b75bf9-54a7-4e16-bf45-3c2d19d5d081 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:12.734735 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b75bf9-54a7-4e16-bf45-3c2d19d5d081 admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:12.735889 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b75bf9-54a7-4e16-bf45-3c2d19d5d081 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:12.738609 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55b75bf9-54a7-4e16-bf45-3c2d19d5d081 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:12.742080 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f497cff5-8494-4683-9510-d4c76645375c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:12.744417 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f497cff5-8494-4683-9510-d4c76645375c admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:12.744874 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f497cff5-8494-4683-9510-d4c76645375c admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:12.746043 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f497cff5-8494-4683-9510-d4c76645375c admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:12.746964 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f497cff5-8494-4683-9510-d4c76645375c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:12.749226 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f497cff5-8494-4683-9510-d4c76645375c admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:12.816766 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1045/2049] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:41:12 2022] POST /identity/v3/users => generated 297 bytes in 225 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:41:12.820929 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b75bf9-54a7-4e16-bf45-3c2d19d5d081 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:12.831679 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55b75bf9-54a7-4e16-bf45-3c2d19d5d081 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:12.836494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6878e327-7c18-4341-9754-86ca714dd388 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:12.837169 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6878e327-7c18-4341-9754-86ca714dd388 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:12.837482 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6878e327-7c18-4341-9754-86ca714dd388 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:12.837766 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6878e327-7c18-4341-9754-86ca714dd388 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:12.838540 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6878e327-7c18-4341-9754-86ca714dd388 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:12.840170 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6878e327-7c18-4341-9754-86ca714dd388 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:12.852326 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1046/2050] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:12 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 12 17:41:12.856458 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6878e327-7c18-4341-9754-86ca714dd388 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:12.866826 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6878e327-7c18-4341-9754-86ca714dd388 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:12.872018 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd9738b-5af6-433c-a4a5-a87da9fd0075 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:12.872764 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd9738b-5af6-433c-a4a5-a87da9fd0075 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:12.873035 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd9738b-5af6-433c-a4a5-a87da9fd0075 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:12.873318 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd9738b-5af6-433c-a4a5-a87da9fd0075 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:12.874089 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd9738b-5af6-433c-a4a5-a87da9fd0075 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:12.876505 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd9738b-5af6-433c-a4a5-a87da9fd0075 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:12.895043 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1005/2051] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:41:12 2022] POST /identity/v3/projects => generated 377 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:41:12.897625 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1047/2052] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:12 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 12 17:41:12.900651 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f497cff5-8494-4683-9510-d4c76645375c admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:12.901373 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd9738b-5af6-433c-a4a5-a87da9fd0075 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:12.928383 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd9738b-5af6-433c-a4a5-a87da9fd0075 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:12.979641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f497cff5-8494-4683-9510-d4c76645375c admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:12.986003 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d5c8bd6-e0de-44ba-97cb-b5dde588991f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:12.986758 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d5c8bd6-e0de-44ba-97cb-b5dde588991f admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:12.987053 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d5c8bd6-e0de-44ba-97cb-b5dde588991f admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:12.987398 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d5c8bd6-e0de-44ba-97cb-b5dde588991f admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:12.988123 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d5c8bd6-e0de-44ba-97cb-b5dde588991f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:12.990160 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d5c8bd6-e0de-44ba-97cb-b5dde588991f admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:13.013470 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309aa2e9-57bd-41dc-85da-50c8f90e2ba3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:13.014635 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309aa2e9-57bd-41dc-85da-50c8f90e2ba3 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:13.014965 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309aa2e9-57bd-41dc-85da-50c8f90e2ba3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:13.015406 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309aa2e9-57bd-41dc-85da-50c8f90e2ba3 admin admin] PATH_INFO: `/v3/projects/a7b5e77559e64538b1b1712f83e90983/users/2a89b85e999a4bf98d3de2b40bcce07c/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:13.032413 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-309aa2e9-57bd-41dc-85da-50c8f90e2ba3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a7b5e77559e64538b1b1712f83e90983, user_id=2a89b85e999a4bf98d3de2b40bcce07c, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:13.035332 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-309aa2e9-57bd-41dc-85da-50c8f90e2ba3 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:13.135317 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1006/2053] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:41:12 2022] POST /identity/v3/users => generated 295 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:41:13.139266 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d5c8bd6-e0de-44ba-97cb-b5dde588991f admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:13.151729 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d5c8bd6-e0de-44ba-97cb-b5dde588991f admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:13.164995 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1048/2054] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:41:12 2022] PUT /identity/v3/projects/a7b5e77559e64538b1b1712f83e90983/users/2a89b85e999a4bf98d3de2b40bcce07c/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:13.188805 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60f4a84-e55c-4b41-ae30-1db4392ab48d None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:13.188805 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60f4a84-e55c-4b41-ae30-1db4392ab48d None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:13.189267 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60f4a84-e55c-4b41-ae30-1db4392ab48d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:13.225946 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-e60f4a84-e55c-4b41-ae30-1db4392ab48d None None] MFA Rules not processed for user `2a89b85e999a4bf98d3de2b40bcce07c`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:13.228221 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-e60f4a84-e55c-4b41-ae30-1db4392ab48d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:13.236654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d79bf9-91b5-4cbf-967a-e9bed43f0d40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:13.238977 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d79bf9-91b5-4cbf-967a-e9bed43f0d40 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:13.239440 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d79bf9-91b5-4cbf-967a-e9bed43f0d40 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:13.240015 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d79bf9-91b5-4cbf-967a-e9bed43f0d40 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:13.241133 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d79bf9-91b5-4cbf-967a-e9bed43f0d40 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:13.245948 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d79bf9-91b5-4cbf-967a-e9bed43f0d40 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:13.268536 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1007/2055] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:13 2022] GET /identity/v3/roles => generated 1420 bytes in 130 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 12 17:41:13.269925 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d79bf9-91b5-4cbf-967a-e9bed43f0d40 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:13.283827 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d79bf9-91b5-4cbf-967a-e9bed43f0d40 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:13.289944 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810217b8-0d8a-4297-9761-af6246deb181 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:13.290912 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810217b8-0d8a-4297-9761-af6246deb181 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:13.291207 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810217b8-0d8a-4297-9761-af6246deb181 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:13.291497 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810217b8-0d8a-4297-9761-af6246deb181 admin admin] PATH_INFO: `/v3/projects/a328dd35ab014be7acdb9605e42b7be4/users/c23af560b5384d6cbae44ed59ee28bbb/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:13.314317 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-810217b8-0d8a-4297-9761-af6246deb181 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a328dd35ab014be7acdb9605e42b7be4, user_id=c23af560b5384d6cbae44ed59ee28bbb, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:13.317428 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-810217b8-0d8a-4297-9761-af6246deb181 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:13.322070 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1049/2056] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:13 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:13.338669 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60f4a84-e55c-4b41-ae30-1db4392ab48d None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:13.338669 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60f4a84-e55c-4b41-ae30-1db4392ab48d None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:13.344567 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d302914c-8ec4-4eb4-ad8d-aa1212f2434f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:13.345441 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d302914c-8ec4-4eb4-ad8d-aa1212f2434f admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:13.345728 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d302914c-8ec4-4eb4-ad8d-aa1212f2434f admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:13.346006 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d302914c-8ec4-4eb4-ad8d-aa1212f2434f admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:13.346795 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d302914c-8ec4-4eb4-ad8d-aa1212f2434f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:13.348735 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d302914c-8ec4-4eb4-ad8d-aa1212f2434f admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:13.432340 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1008/2057] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:41:13 2022] PUT /identity/v3/projects/a328dd35ab014be7acdb9605e42b7be4/users/c23af560b5384d6cbae44ed59ee28bbb/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:13.439552 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810217b8-0d8a-4297-9761-af6246deb181 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:13.489176 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1050/2058] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:41:13 2022] POST /identity/v3/projects => generated 393 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:41:13.493484 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d302914c-8ec4-4eb4-ad8d-aa1212f2434f admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:13.530080 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810217b8-0d8a-4297-9761-af6246deb181 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:13.595658 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d302914c-8ec4-4eb4-ad8d-aa1212f2434f admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:13.603670 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25be1483-b16d-484e-a91d-e70fcefec4f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:13.609868 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25be1483-b16d-484e-a91d-e70fcefec4f8 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:13.609868 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25be1483-b16d-484e-a91d-e70fcefec4f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:13.609868 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25be1483-b16d-484e-a91d-e70fcefec4f8 admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:13.609868 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25be1483-b16d-484e-a91d-e70fcefec4f8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:13.609868 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25be1483-b16d-484e-a91d-e70fcefec4f8 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:13.616501 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb0951e-bda1-4a27-8023-d4d559c95ac5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:13.617361 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb0951e-bda1-4a27-8023-d4d559c95ac5 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:13.617725 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb0951e-bda1-4a27-8023-d4d559c95ac5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:13.618182 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb0951e-bda1-4a27-8023-d4d559c95ac5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:13.619169 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb0951e-bda1-4a27-8023-d4d559c95ac5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:13.621000 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb0951e-bda1-4a27-8023-d4d559c95ac5 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:13.632968 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1009/2059] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:13 2022] GET /identity/v3/roles => generated 1420 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:41:13.637113 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb0951e-bda1-4a27-8023-d4d559c95ac5 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:13.648286 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb0951e-bda1-4a27-8023-d4d559c95ac5 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:13.653487 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16199029-34a2-4c55-af8a-66d204f10971 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:13.654415 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16199029-34a2-4c55-af8a-66d204f10971 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:13.654742 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16199029-34a2-4c55-af8a-66d204f10971 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:13.655052 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16199029-34a2-4c55-af8a-66d204f10971 admin admin] PATH_INFO: `/v3/projects/a328dd35ab014be7acdb9605e42b7be4/users/c23af560b5384d6cbae44ed59ee28bbb/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:13.657841 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16199029-34a2-4c55-af8a-66d204f10971 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a328dd35ab014be7acdb9605e42b7be4, user_id=c23af560b5384d6cbae44ed59ee28bbb, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:13.660783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16199029-34a2-4c55-af8a-66d204f10971 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:13.673990 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1010/2060] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:41:13 2022] PUT /identity/v3/projects/a328dd35ab014be7acdb9605e42b7be4/users/c23af560b5384d6cbae44ed59ee28bbb/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:41:13.721258 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1051/2061] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:41:13 2022] POST /identity/v3/users => generated 303 bytes in 228 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:41:13.725293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16199029-34a2-4c55-af8a-66d204f10971 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:13.808918 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16199029-34a2-4c55-af8a-66d204f10971 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:13.815184 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc4534f-6540-4357-909b-9a5f4531bd85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:13.815897 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc4534f-6540-4357-909b-9a5f4531bd85 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:13.816248 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc4534f-6540-4357-909b-9a5f4531bd85 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:13.816543 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc4534f-6540-4357-909b-9a5f4531bd85 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:13.817535 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc4534f-6540-4357-909b-9a5f4531bd85 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:13.819402 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc4534f-6540-4357-909b-9a5f4531bd85 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:13.834313 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1011/2062] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:13 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:41:13.835606 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25be1483-b16d-484e-a91d-e70fcefec4f8 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:13.849546 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25be1483-b16d-484e-a91d-e70fcefec4f8 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:13.863400 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076203a7-1258-4419-aa80-cfcaf0682a90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:13.865246 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076203a7-1258-4419-aa80-cfcaf0682a90 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:13.865765 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076203a7-1258-4419-aa80-cfcaf0682a90 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:13.866289 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076203a7-1258-4419-aa80-cfcaf0682a90 admin admin] PATH_INFO: `/v3/projects/f14a7cdb538d4c40ac9f7a744e2ea940/users/5c7c7db7f64744baaa786adbb14b1dfd/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:13.893284 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-076203a7-1258-4419-aa80-cfcaf0682a90 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f14a7cdb538d4c40ac9f7a744e2ea940, user_id=5c7c7db7f64744baaa786adbb14b1dfd, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:13.896329 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-076203a7-1258-4419-aa80-cfcaf0682a90 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:14.076036 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1052/2063] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:41:13 2022] PUT /identity/v3/projects/f14a7cdb538d4c40ac9f7a744e2ea940/users/5c7c7db7f64744baaa786adbb14b1dfd/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:14.080998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc4534f-6540-4357-909b-9a5f4531bd85 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:14.094804 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc4534f-6540-4357-909b-9a5f4531bd85 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:14.196060 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8be22b-05a0-4a51-b2fa-6a0f21031e55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:14.196928 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8be22b-05a0-4a51-b2fa-6a0f21031e55 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:14.197284 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8be22b-05a0-4a51-b2fa-6a0f21031e55 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:14.202480 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8be22b-05a0-4a51-b2fa-6a0f21031e55 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:14.202480 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed8be22b-05a0-4a51-b2fa-6a0f21031e55 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:14.202480 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed8be22b-05a0-4a51-b2fa-6a0f21031e55 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:14.222391 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1012/2064] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:14 2022] GET /identity/v3/roles => generated 1420 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 12 17:41:14.226546 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076203a7-1258-4419-aa80-cfcaf0682a90 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:14.245732 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076203a7-1258-4419-aa80-cfcaf0682a90 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:14.252044 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bd7ebc-f113-48c5-95a0-3d61d7228647 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:14.253459 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bd7ebc-f113-48c5-95a0-3d61d7228647 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:14.253944 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bd7ebc-f113-48c5-95a0-3d61d7228647 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:14.255335 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bd7ebc-f113-48c5-95a0-3d61d7228647 admin admin] PATH_INFO: `/v3/projects/f14a7cdb538d4c40ac9f7a744e2ea940/users/5c7c7db7f64744baaa786adbb14b1dfd/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:14.257947 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7bd7ebc-f113-48c5-95a0-3d61d7228647 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f14a7cdb538d4c40ac9f7a744e2ea940, user_id=5c7c7db7f64744baaa786adbb14b1dfd, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:14.266299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7bd7ebc-f113-48c5-95a0-3d61d7228647 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:14.282421 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1053/2065] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:41:14 2022] PUT /identity/v3/projects/f14a7cdb538d4c40ac9f7a744e2ea940/users/5c7c7db7f64744baaa786adbb14b1dfd/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:14.292353 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50568ed9-2387-4444-ae46-8cc945fb5aba None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:14.293562 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50568ed9-2387-4444-ae46-8cc945fb5aba None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:14.294032 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50568ed9-2387-4444-ae46-8cc945fb5aba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:14.361794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-50568ed9-2387-4444-ae46-8cc945fb5aba None None] MFA Rules not processed for user `5c7c7db7f64744baaa786adbb14b1dfd`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:14.363417 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-50568ed9-2387-4444-ae46-8cc945fb5aba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:14.507064 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1013/2066] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:14 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:14.584088 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bd7ebc-f113-48c5-95a0-3d61d7228647 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:14.586327 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-e7bd7ebc-f113-48c5-95a0-3d61d7228647 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:14.632720 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:14.784370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bd7ebc-f113-48c5-95a0-3d61d7228647 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:14.791363 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4caa02e4-7527-4de9-b09d-52d7dc5afc99 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:14.792058 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4caa02e4-7527-4de9-b09d-52d7dc5afc99 service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:14.792354 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4caa02e4-7527-4de9-b09d-52d7dc5afc99 service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:14.792643 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4caa02e4-7527-4de9-b09d-52d7dc5afc99 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:14.815556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4caa02e4-7527-4de9-b09d-52d7dc5afc99 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:14.817040 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4caa02e4-7527-4de9-b09d-52d7dc5afc99 service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:14.839913 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1054/2067] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:41:14 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 256 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:41:14.892403 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:15.014601 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50568ed9-2387-4444-ae46-8cc945fb5aba None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:15.027021 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50568ed9-2387-4444-ae46-8cc945fb5aba None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:15.036159 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bec0f8d-849a-41b1-9baf-289dcc4cc347 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:15.036953 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bec0f8d-849a-41b1-9baf-289dcc4cc347 service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:15.037838 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bec0f8d-849a-41b1-9baf-289dcc4cc347 service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:15.038134 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bec0f8d-849a-41b1-9baf-289dcc4cc347 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:15.185714 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bec0f8d-849a-41b1-9baf-289dcc4cc347 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:15.188159 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bec0f8d-849a-41b1-9baf-289dcc4cc347 service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:15.214353 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1014/2068] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:41:15 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 201 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:41:16.999067 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:17.786574 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:19.004427 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:20.554372 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4caa02e4-7527-4de9-b09d-52d7dc5afc99 service cinder] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:20.557776 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-4caa02e4-7527-4de9-b09d-52d7dc5afc99 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:20.713891 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4caa02e4-7527-4de9-b09d-52d7dc5afc99 service cinder] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:20.721341 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7d6270-2180-4385-9653-4807668da4b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:20.722830 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7d6270-2180-4385-9653-4807668da4b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:20.723484 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7d6270-2180-4385-9653-4807668da4b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:20.724107 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7d6270-2180-4385-9653-4807668da4b3 admin admin] PATH_INFO: `/v3/users/c3d16229c134464d8326bdd874f7c180` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:20.726857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7d6270-2180-4385-9653-4807668da4b3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c3d16229c134464d8326bdd874f7c180)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:20.729653 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7d6270-2180-4385-9653-4807668da4b3 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:20.978069 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-cf7d6270-2180-4385-9653-4807668da4b3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c3d16229c134464d8326bdd874f7c180'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:20.979665 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-cf7d6270-2180-4385-9653-4807668da4b3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c3d16229c134464d8326bdd874f7c180'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:20.980945 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-cf7d6270-2180-4385-9653-4807668da4b3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c3d16229c134464d8326bdd874f7c180'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:20.991907 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-cf7d6270-2180-4385-9653-4807668da4b3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c3d16229c134464d8326bdd874f7c180'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:21.011944 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=89,keystone:INSERT=2,keystone:DELETE=9 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:21.013478 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-cf7d6270-2180-4385-9653-4807668da4b3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c3d16229c134464d8326bdd874f7c180'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:21.113996 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1055/2069] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:41:20 2022] DELETE /identity/v3/users/c3d16229c134464d8326bdd874f7c180 => generated 0 bytes in 560 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:21.119381 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e11d73-f4d3-4ef1-a092-7d0bdc798548 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:21.120157 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e11d73-f4d3-4ef1-a092-7d0bdc798548 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:21.120157 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e11d73-f4d3-4ef1-a092-7d0bdc798548 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:21.157563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-b3e11d73-f4d3-4ef1-a092-7d0bdc798548 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:21.159728 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-b3e11d73-f4d3-4ef1-a092-7d0bdc798548 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:21.282288 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1015/2070] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:21.292552 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7d6270-2180-4385-9653-4807668da4b3 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:21.294204 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-cf7d6270-2180-4385-9653-4807668da4b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:21.393899 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7d6270-2180-4385-9653-4807668da4b3 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:21.399069 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06b0896-736f-438b-9a25-fbe5d884a583 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:21.399743 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06b0896-736f-438b-9a25-fbe5d884a583 service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:21.400104 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06b0896-736f-438b-9a25-fbe5d884a583 service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:21.400456 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06b0896-736f-438b-9a25-fbe5d884a583 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:21.436455 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06b0896-736f-438b-9a25-fbe5d884a583 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:21.438926 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06b0896-736f-438b-9a25-fbe5d884a583 service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:21.460172 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1056/2071] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:41:21 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:41:21.529065 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e11d73-f4d3-4ef1-a092-7d0bdc798548 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:21.530666 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-b3e11d73-f4d3-4ef1-a092-7d0bdc798548 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:21.545402 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e11d73-f4d3-4ef1-a092-7d0bdc798548 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:21.552726 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df7d057-a681-4876-8ab8-8321acbe2413 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:21.553680 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df7d057-a681-4876-8ab8-8321acbe2413 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:21.554204 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df7d057-a681-4876-8ab8-8321acbe2413 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:21.554703 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df7d057-a681-4876-8ab8-8321acbe2413 admin admin] PATH_INFO: `/v3/users/2a89b85e999a4bf98d3de2b40bcce07c` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:21.556296 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df7d057-a681-4876-8ab8-8321acbe2413 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2a89b85e999a4bf98d3de2b40bcce07c)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:21.559210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df7d057-a681-4876-8ab8-8321acbe2413 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:21.884901 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-4df7d057-a681-4876-8ab8-8321acbe2413 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2a89b85e999a4bf98d3de2b40bcce07c'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:21.981144 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-4df7d057-a681-4876-8ab8-8321acbe2413 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2a89b85e999a4bf98d3de2b40bcce07c'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:21.982067 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-4df7d057-a681-4876-8ab8-8321acbe2413 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2a89b85e999a4bf98d3de2b40bcce07c'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:21.995401 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-4df7d057-a681-4876-8ab8-8321acbe2413 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2a89b85e999a4bf98d3de2b40bcce07c'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:22.069581 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-4df7d057-a681-4876-8ab8-8321acbe2413 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2a89b85e999a4bf98d3de2b40bcce07c'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:22.086876 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1016/2072] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:41:21 2022] DELETE /identity/v3/users/2a89b85e999a4bf98d3de2b40bcce07c => generated 0 bytes in 558 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:22.091863 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05e90ee-41d5-47b6-a2e6-510197546231 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:22.092432 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05e90ee-41d5-47b6-a2e6-510197546231 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:22.092656 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05e90ee-41d5-47b6-a2e6-510197546231 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:22.145359 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-e05e90ee-41d5-47b6-a2e6-510197546231 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:22.147402 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-e05e90ee-41d5-47b6-a2e6-510197546231 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:22.242230 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1057/2073] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:41:22.264630 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df7d057-a681-4876-8ab8-8321acbe2413 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:22.264630 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-4df7d057-a681-4876-8ab8-8321acbe2413 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:22.269955 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05e90ee-41d5-47b6-a2e6-510197546231 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:22.272185 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-e05e90ee-41d5-47b6-a2e6-510197546231 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:22.290331 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05e90ee-41d5-47b6-a2e6-510197546231 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:22.317367 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2cff871-2c0d-4698-afa8-3df8f8c3038f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:22.318964 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cff871-2c0d-4698-afa8-3df8f8c3038f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:22.323378 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cff871-2c0d-4698-afa8-3df8f8c3038f admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:22.337812 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cff871-2c0d-4698-afa8-3df8f8c3038f admin admin] PATH_INFO: `/v3/projects/7a4e997384574368b02ae1c38be768e8` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:22.343562 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2cff871-2c0d-4698-afa8-3df8f8c3038f admin admin] RBAC: Authorizing `identity:delete_project(project_id=7a4e997384574368b02ae1c38be768e8)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:22.352047 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2cff871-2c0d-4698-afa8-3df8f8c3038f admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:22.464687 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df7d057-a681-4876-8ab8-8321acbe2413 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:22.471371 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3321b48-7be9-4baf-8b18-301557d6f8c2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:22.472930 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3321b48-7be9-4baf-8b18-301557d6f8c2 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:22.473571 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3321b48-7be9-4baf-8b18-301557d6f8c2 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:22.474675 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3321b48-7be9-4baf-8b18-301557d6f8c2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:22.494010 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3321b48-7be9-4baf-8b18-301557d6f8c2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:22.501022 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3321b48-7be9-4baf-8b18-301557d6f8c2 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:22.539464 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1017/2074] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:41:22 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 278 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:41:22.545729 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf411b5e-7128-437f-925c-1736a6cf2670 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:22.546074 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf411b5e-7128-437f-925c-1736a6cf2670 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:22.548538 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf411b5e-7128-437f-925c-1736a6cf2670 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:22.624392 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e2cff871-2c0d-4698-afa8-3df8f8c3038f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7a4e997384574368b02ae1c38be768e8', 'request_id': 'req-e2cff871-2c0d-4698-afa8-3df8f8c3038f'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:22.649851 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:22.650350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e2cff871-2c0d-4698-afa8-3df8f8c3038f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7a4e997384574368b02ae1c38be768e8', 'request_id': 'req-e2cff871-2c0d-4698-afa8-3df8f8c3038f'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:22.665976 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-bf411b5e-7128-437f-925c-1736a6cf2670 None None] MFA Rules not processed for user `c23af560b5384d6cbae44ed59ee28bbb`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:22.667321 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-bf411b5e-7128-437f-925c-1736a6cf2670 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:22.730064 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e2cff871-2c0d-4698-afa8-3df8f8c3038f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7a4e997384574368b02ae1c38be768e8', 'request_id': 'req-e2cff871-2c0d-4698-afa8-3df8f8c3038f'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:22.751550 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e2cff871-2c0d-4698-afa8-3df8f8c3038f admin admin] The token cache is being invalidate because project 7a4e997384574368b02ae1c38be768e8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:41:22.752265 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e2cff871-2c0d-4698-afa8-3df8f8c3038f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:22.753525 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e2cff871-2c0d-4698-afa8-3df8f8c3038f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:22.756551 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1058/2075] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:41:22 2022] DELETE /identity/v3/projects/7a4e997384574368b02ae1c38be768e8 => generated 0 bytes in 487 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:22.760419 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2cff871-2c0d-4698-afa8-3df8f8c3038f admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:22.764636 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-e2cff871-2c0d-4698-afa8-3df8f8c3038f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:22.816862 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=90,keystone:INSERT=2,keystone:DELETE=8 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:22.946762 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1018/2076] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:22 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 407 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:22.961025 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2cff871-2c0d-4698-afa8-3df8f8c3038f admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:22.971579 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf411b5e-7128-437f-925c-1736a6cf2670 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:22.974550 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b168eef5-0f5c-40d3-8375-4775f09a9548 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:22.975042 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-bf411b5e-7128-437f-925c-1736a6cf2670 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:22.975493 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b168eef5-0f5c-40d3-8375-4775f09a9548 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:22.975792 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b168eef5-0f5c-40d3-8375-4775f09a9548 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:22.976073 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b168eef5-0f5c-40d3-8375-4775f09a9548 admin admin] PATH_INFO: `/v3/users/ef296a8ff467471abc46b6f80f16287c` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:22.981699 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b168eef5-0f5c-40d3-8375-4775f09a9548 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ef296a8ff467471abc46b6f80f16287c)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:22.985913 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b168eef5-0f5c-40d3-8375-4775f09a9548 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:23.140590 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf411b5e-7128-437f-925c-1736a6cf2670 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:23.147441 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db6f4ef-9f52-4d25-af46-a01a648ea8a2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:23.148130 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db6f4ef-9f52-4d25-af46-a01a648ea8a2 service nova] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:23.148419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db6f4ef-9f52-4d25-af46-a01a648ea8a2 service nova] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:23.148700 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db6f4ef-9f52-4d25-af46-a01a648ea8a2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:23.150794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-2db6f4ef-9f52-4d25-af46-a01a648ea8a2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:23.335919 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db6f4ef-9f52-4d25-af46-a01a648ea8a2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:23.336654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db6f4ef-9f52-4d25-af46-a01a648ea8a2 service nova] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:23.350943 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1019/2077] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:41:22 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:41:23.358950 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-b168eef5-0f5c-40d3-8375-4775f09a9548 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ef296a8ff467471abc46b6f80f16287c'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:23.360838 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-b168eef5-0f5c-40d3-8375-4775f09a9548 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ef296a8ff467471abc46b6f80f16287c'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:23.363482 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-b168eef5-0f5c-40d3-8375-4775f09a9548 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ef296a8ff467471abc46b6f80f16287c'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:23.389278 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-b168eef5-0f5c-40d3-8375-4775f09a9548 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ef296a8ff467471abc46b6f80f16287c'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:23.417704 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-b168eef5-0f5c-40d3-8375-4775f09a9548 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ef296a8ff467471abc46b6f80f16287c'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:23.554910 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1059/2078] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:41:22 2022] DELETE /identity/v3/users/ef296a8ff467471abc46b6f80f16287c => generated 0 bytes in 795 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:23.712655 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db6f4ef-9f52-4d25-af46-a01a648ea8a2 service nova] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:23.715451 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-2db6f4ef-9f52-4d25-af46-a01a648ea8a2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:23.744818 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db6f4ef-9f52-4d25-af46-a01a648ea8a2 service nova] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:23.907459 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d94d57a-cb25-4b55-bf2f-1580bd397534 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:23.908631 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d94d57a-cb25-4b55-bf2f-1580bd397534 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:23.908960 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d94d57a-cb25-4b55-bf2f-1580bd397534 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:23.909396 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d94d57a-cb25-4b55-bf2f-1580bd397534 admin admin] PATH_INFO: `/v3/projects/a7b5e77559e64538b1b1712f83e90983` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:23.911076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d94d57a-cb25-4b55-bf2f-1580bd397534 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a7b5e77559e64538b1b1712f83e90983)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:23.913279 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d94d57a-cb25-4b55-bf2f-1580bd397534 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:24.102851 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-4d94d57a-cb25-4b55-bf2f-1580bd397534 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a7b5e77559e64538b1b1712f83e90983', 'request_id': 'req-4d94d57a-cb25-4b55-bf2f-1580bd397534'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:24.112381 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-4d94d57a-cb25-4b55-bf2f-1580bd397534 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a7b5e77559e64538b1b1712f83e90983', 'request_id': 'req-4d94d57a-cb25-4b55-bf2f-1580bd397534'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:24.121759 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-4d94d57a-cb25-4b55-bf2f-1580bd397534 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a7b5e77559e64538b1b1712f83e90983', 'request_id': 'req-4d94d57a-cb25-4b55-bf2f-1580bd397534'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:24.169630 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:24.220018 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-4d94d57a-cb25-4b55-bf2f-1580bd397534 admin admin] The token cache is being invalidate because project a7b5e77559e64538b1b1712f83e90983 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:41:24.220386 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-4d94d57a-cb25-4b55-bf2f-1580bd397534 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:24.221936 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-4d94d57a-cb25-4b55-bf2f-1580bd397534 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:24.224192 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1020/2079] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:41:23 2022] DELETE /identity/v3/projects/a7b5e77559e64538b1b1712f83e90983 => generated 0 bytes in 512 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:24.233370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b168eef5-0f5c-40d3-8375-4775f09a9548 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:24.234910 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-b168eef5-0f5c-40d3-8375-4775f09a9548 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:24.255872 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b168eef5-0f5c-40d3-8375-4775f09a9548 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:24.364635 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14955431-6b8d-46b7-8f80-c37aefe5a088 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:24.365710 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14955431-6b8d-46b7-8f80-c37aefe5a088 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:24.366292 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14955431-6b8d-46b7-8f80-c37aefe5a088 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:24.366742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14955431-6b8d-46b7-8f80-c37aefe5a088 admin admin] PATH_INFO: `/v3/users/5c7c7db7f64744baaa786adbb14b1dfd` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:24.368746 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14955431-6b8d-46b7-8f80-c37aefe5a088 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5c7c7db7f64744baaa786adbb14b1dfd)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:24.371832 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14955431-6b8d-46b7-8f80-c37aefe5a088 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:24.595335 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d94d57a-cb25-4b55-bf2f-1580bd397534 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:24.597724 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-4d94d57a-cb25-4b55-bf2f-1580bd397534 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:24.617817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d94d57a-cb25-4b55-bf2f-1580bd397534 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:24.624452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c37e6e-1520-47b8-bcae-9a49fb0cb373 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:24.627425 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c37e6e-1520-47b8-bcae-9a49fb0cb373 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:24.628107 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c37e6e-1520-47b8-bcae-9a49fb0cb373 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:24.628538 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c37e6e-1520-47b8-bcae-9a49fb0cb373 admin admin] PATH_INFO: `/v3/projects/9e8ead470193456d8d56a23791099482` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:24.630113 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c37e6e-1520-47b8-bcae-9a49fb0cb373 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9e8ead470193456d8d56a23791099482)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:24.633882 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c37e6e-1520-47b8-bcae-9a49fb0cb373 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:24.669822 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:24.689304 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-14955431-6b8d-46b7-8f80-c37aefe5a088 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5c7c7db7f64744baaa786adbb14b1dfd'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:24.692472 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-14955431-6b8d-46b7-8f80-c37aefe5a088 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5c7c7db7f64744baaa786adbb14b1dfd'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:24.693313 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-14955431-6b8d-46b7-8f80-c37aefe5a088 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5c7c7db7f64744baaa786adbb14b1dfd'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:24.701918 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-14955431-6b8d-46b7-8f80-c37aefe5a088 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5c7c7db7f64744baaa786adbb14b1dfd'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:24.716095 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-14955431-6b8d-46b7-8f80-c37aefe5a088 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5c7c7db7f64744baaa786adbb14b1dfd'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:24.814068 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-97c37e6e-1520-47b8-bcae-9a49fb0cb373 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9e8ead470193456d8d56a23791099482', 'request_id': 'req-97c37e6e-1520-47b8-bcae-9a49fb0cb373'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:24.828577 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-97c37e6e-1520-47b8-bcae-9a49fb0cb373 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9e8ead470193456d8d56a23791099482', 'request_id': 'req-97c37e6e-1520-47b8-bcae-9a49fb0cb373'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:24.837854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-97c37e6e-1520-47b8-bcae-9a49fb0cb373 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9e8ead470193456d8d56a23791099482', 'request_id': 'req-97c37e6e-1520-47b8-bcae-9a49fb0cb373'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:24.856776 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1060/2080] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:41:24 2022] DELETE /identity/v3/users/5c7c7db7f64744baaa786adbb14b1dfd => generated 0 bytes in 624 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:25.094979 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-97c37e6e-1520-47b8-bcae-9a49fb0cb373 admin admin] The token cache is being invalidate because project 9e8ead470193456d8d56a23791099482 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:41:25.095518 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-97c37e6e-1520-47b8-bcae-9a49fb0cb373 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:25.096663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-97c37e6e-1520-47b8-bcae-9a49fb0cb373 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:25.105950 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1021/2081] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:41:24 2022] DELETE /identity/v3/projects/9e8ead470193456d8d56a23791099482 => generated 0 bytes in 512 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:25.113915 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce907652-b62d-4a6d-8be8-ea7b0da907f7 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:25.114920 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce907652-b62d-4a6d-8be8-ea7b0da907f7 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:25.115416 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce907652-b62d-4a6d-8be8-ea7b0da907f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:25.144481 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-ce907652-b62d-4a6d-8be8-ea7b0da907f7 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:25.146474 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-ce907652-b62d-4a6d-8be8-ea7b0da907f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:25.225379 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1061/2082] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:25.239133 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179619fe-b211-4fb8-9569-131d9a89a453 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:25.239725 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179619fe-b211-4fb8-9569-131d9a89a453 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:25.240308 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179619fe-b211-4fb8-9569-131d9a89a453 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:25.280148 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-179619fe-b211-4fb8-9569-131d9a89a453 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:25.286939 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-179619fe-b211-4fb8-9569-131d9a89a453 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:25.331670 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1022/2083] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:25.336613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce907652-b62d-4a6d-8be8-ea7b0da907f7 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:25.366603 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce907652-b62d-4a6d-8be8-ea7b0da907f7 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:25.379146 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e38f20-06cc-4499-8b18-bea04312d4bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:25.380239 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e38f20-06cc-4499-8b18-bea04312d4bb admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:25.382615 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e38f20-06cc-4499-8b18-bea04312d4bb admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:25.382615 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e38f20-06cc-4499-8b18-bea04312d4bb admin admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:25.382615 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16e38f20-06cc-4499-8b18-bea04312d4bb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:41:25.382615 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16e38f20-06cc-4499-8b18-bea04312d4bb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:25.384496 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16e38f20-06cc-4499-8b18-bea04312d4bb admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:25.407493 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1062/2084] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:41:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:41:25.412965 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-179619fe-b211-4fb8-9569-131d9a89a453 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:25.426465 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-179619fe-b211-4fb8-9569-131d9a89a453 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:25.432697 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337165e0-437d-4aab-9fb5-31f45ea836cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:25.433452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337165e0-437d-4aab-9fb5-31f45ea836cb admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:25.433748 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337165e0-437d-4aab-9fb5-31f45ea836cb admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:25.434074 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337165e0-437d-4aab-9fb5-31f45ea836cb admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:25.434849 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337165e0-437d-4aab-9fb5-31f45ea836cb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:25.436888 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337165e0-437d-4aab-9fb5-31f45ea836cb admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:25.611301 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1023/2085] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:41:25 2022] POST /identity/v3/projects => generated 367 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:41:25.616629 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e38f20-06cc-4499-8b18-bea04312d4bb admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:25.632229 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e38f20-06cc-4499-8b18-bea04312d4bb admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:25.693311 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337165e0-437d-4aab-9fb5-31f45ea836cb admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:25.696038 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-337165e0-437d-4aab-9fb5-31f45ea836cb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:25.741457 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:25.742916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f4a3fdd-2c8a-4ce0-b21c-1d0e0ea1d1d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:25.743622 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4a3fdd-2c8a-4ce0-b21c-1d0e0ea1d1d3 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:25.743901 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4a3fdd-2c8a-4ce0-b21c-1d0e0ea1d1d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:25.747064 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4a3fdd-2c8a-4ce0-b21c-1d0e0ea1d1d3 admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:25.748166 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f4a3fdd-2c8a-4ce0-b21c-1d0e0ea1d1d3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:25.751751 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f4a3fdd-2c8a-4ce0-b21c-1d0e0ea1d1d3 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:25.823293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337165e0-437d-4aab-9fb5-31f45ea836cb admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:25.829918 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d5fb95-887f-4606-8a7d-b86c31ece902 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:25.831103 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d5fb95-887f-4606-8a7d-b86c31ece902 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:25.831397 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d5fb95-887f-4606-8a7d-b86c31ece902 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:25.835932 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d5fb95-887f-4606-8a7d-b86c31ece902 admin admin] PATH_INFO: `/v3/projects/f14a7cdb538d4c40ac9f7a744e2ea940` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:25.835932 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d5fb95-887f-4606-8a7d-b86c31ece902 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f14a7cdb538d4c40ac9f7a744e2ea940)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:25.835932 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d5fb95-887f-4606-8a7d-b86c31ece902 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:25.862065 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1063/2086] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:41:25 2022] POST /identity/v3/users => generated 284 bytes in 245 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:41:25.865661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f4a3fdd-2c8a-4ce0-b21c-1d0e0ea1d1d3 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:25.876400 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f4a3fdd-2c8a-4ce0-b21c-1d0e0ea1d1d3 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:25.882150 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6b1458-1cd8-46fc-a542-6332b071122d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:25.882931 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6b1458-1cd8-46fc-a542-6332b071122d admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:25.883720 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6b1458-1cd8-46fc-a542-6332b071122d admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:25.883720 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6b1458-1cd8-46fc-a542-6332b071122d admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:25.884242 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6b1458-1cd8-46fc-a542-6332b071122d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:25.887278 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6b1458-1cd8-46fc-a542-6332b071122d admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:25.912175 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1064/2087] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:25 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 12 17:41:25.916559 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6b1458-1cd8-46fc-a542-6332b071122d admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:25.938627 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6b1458-1cd8-46fc-a542-6332b071122d admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:25.971007 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09418169-1ceb-4bf6-b74d-e2a3716debc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:25.972334 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09418169-1ceb-4bf6-b74d-e2a3716debc7 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:25.972628 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09418169-1ceb-4bf6-b74d-e2a3716debc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:25.973348 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09418169-1ceb-4bf6-b74d-e2a3716debc7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:25.974121 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09418169-1ceb-4bf6-b74d-e2a3716debc7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:25.976648 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09418169-1ceb-4bf6-b74d-e2a3716debc7 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:26.009427 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1065/2088] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:25 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:41:26.014326 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09418169-1ceb-4bf6-b74d-e2a3716debc7 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:26.015658 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-24d5fb95-887f-4606-8a7d-b86c31ece902 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f14a7cdb538d4c40ac9f7a744e2ea940', 'request_id': 'req-24d5fb95-887f-4606-8a7d-b86c31ece902'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:26.029237 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-24d5fb95-887f-4606-8a7d-b86c31ece902 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f14a7cdb538d4c40ac9f7a744e2ea940', 'request_id': 'req-24d5fb95-887f-4606-8a7d-b86c31ece902'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:26.038920 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09418169-1ceb-4bf6-b74d-e2a3716debc7 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:26.047605 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-24d5fb95-887f-4606-8a7d-b86c31ece902 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f14a7cdb538d4c40ac9f7a744e2ea940', 'request_id': 'req-24d5fb95-887f-4606-8a7d-b86c31ece902'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:26.190538 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-24d5fb95-887f-4606-8a7d-b86c31ece902 admin admin] The token cache is being invalidate because project f14a7cdb538d4c40ac9f7a744e2ea940 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:41:26.190538 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-24d5fb95-887f-4606-8a7d-b86c31ece902 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:26.190538 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-24d5fb95-887f-4606-8a7d-b86c31ece902 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:26.195665 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1024/2089] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:41:25 2022] DELETE /identity/v3/projects/f14a7cdb538d4c40ac9f7a744e2ea940 => generated 0 bytes in 500 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:26.198274 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f7aae8-788b-46a3-8ca9-e2a3d4a1fce8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:26.200001 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632f226f-b70a-43ff-a97c-494c253e01cb None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:26.203240 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632f226f-b70a-43ff-a97c-494c253e01cb None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:26.203687 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632f226f-b70a-43ff-a97c-494c253e01cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:26.204869 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f7aae8-788b-46a3-8ca9-e2a3d4a1fce8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:26.205413 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f7aae8-788b-46a3-8ca9-e2a3d4a1fce8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:26.205907 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f7aae8-788b-46a3-8ca9-e2a3d4a1fce8 admin admin] PATH_INFO: `/v3/projects/884c4406f6de4c4fbb56866fdad40575/users/1dc3556822484d24967f2288813c8bee/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:26.267908 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-632f226f-b70a-43ff-a97c-494c253e01cb None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:26.269502 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-632f226f-b70a-43ff-a97c-494c253e01cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:26.271659 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71f7aae8-788b-46a3-8ca9-e2a3d4a1fce8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=884c4406f6de4c4fbb56866fdad40575, user_id=1dc3556822484d24967f2288813c8bee, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:26.274966 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71f7aae8-788b-46a3-8ca9-e2a3d4a1fce8 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:26.285231 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1025/2090] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:26.299024 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae6e935-e421-4340-b240-cdce6ea4ae2b None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:26.299599 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae6e935-e421-4340-b240-cdce6ea4ae2b None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:26.300032 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae6e935-e421-4340-b240-cdce6ea4ae2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:26.338564 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-1ae6e935-e421-4340-b240-cdce6ea4ae2b None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:26.341926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-1ae6e935-e421-4340-b240-cdce6ea4ae2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:26.363801 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1026/2091] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 67 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:26.369102 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae6e935-e421-4340-b240-cdce6ea4ae2b None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:26.379508 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae6e935-e421-4340-b240-cdce6ea4ae2b None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:26.393343 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2786c59e-6e35-42c8-aaac-06f48f3248e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:26.399096 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2786c59e-6e35-42c8-aaac-06f48f3248e8 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:26.399096 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2786c59e-6e35-42c8-aaac-06f48f3248e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:26.399096 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2786c59e-6e35-42c8-aaac-06f48f3248e8 admin admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:26.399416 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1066/2092] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:41:26 2022] PUT /identity/v3/projects/884c4406f6de4c4fbb56866fdad40575/users/1dc3556822484d24967f2288813c8bee/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 383 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:26.399830 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2786c59e-6e35-42c8-aaac-06f48f3248e8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:41:26.399830 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2786c59e-6e35-42c8-aaac-06f48f3248e8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:26.401219 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2786c59e-6e35-42c8-aaac-06f48f3248e8 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:26.407779 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ab7fd0-ca7e-4951-8cc3-f3f0a8aa9626 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:26.408173 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ab7fd0-ca7e-4951-8cc3-f3f0a8aa9626 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:26.408535 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ab7fd0-ca7e-4951-8cc3-f3f0a8aa9626 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:26.424364 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1027/2093] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:41:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:41:26.426815 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2786c59e-6e35-42c8-aaac-06f48f3248e8 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:26.449804 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2786c59e-6e35-42c8-aaac-06f48f3248e8 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:26.503157 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-62ab7fd0-ca7e-4951-8cc3-f3f0a8aa9626 None None] MFA Rules not processed for user `1dc3556822484d24967f2288813c8bee`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:26.513647 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-62ab7fd0-ca7e-4951-8cc3-f3f0a8aa9626 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:26.545165 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:26.561045 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e47a05-f0da-412a-b788-c70a70228007 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:26.562102 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e47a05-f0da-412a-b788-c70a70228007 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:26.563253 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e47a05-f0da-412a-b788-c70a70228007 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:26.563946 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e47a05-f0da-412a-b788-c70a70228007 admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:26.567181 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e47a05-f0da-412a-b788-c70a70228007 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:26.570872 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e47a05-f0da-412a-b788-c70a70228007 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:26.650608 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1067/2094] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:26 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 244 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:41:26.655830 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ab7fd0-ca7e-4951-8cc3-f3f0a8aa9626 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:26.657485 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-62ab7fd0-ca7e-4951-8cc3-f3f0a8aa9626 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:26.671256 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ab7fd0-ca7e-4951-8cc3-f3f0a8aa9626 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:26.679746 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66776553-6e9a-464e-8817-3826214c1f94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:26.680255 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66776553-6e9a-464e-8817-3826214c1f94 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:26.680706 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66776553-6e9a-464e-8817-3826214c1f94 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:26.681254 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66776553-6e9a-464e-8817-3826214c1f94 admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:26.682023 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66776553-6e9a-464e-8817-3826214c1f94 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:26.684538 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66776553-6e9a-464e-8817-3826214c1f94 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:26.715164 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1028/2095] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:41:26 2022] POST /identity/v3/projects => generated 383 bytes in 289 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:41:26.719947 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e47a05-f0da-412a-b788-c70a70228007 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:26.814196 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e47a05-f0da-412a-b788-c70a70228007 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:26.821629 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4c261b-4d7d-4abb-a996-f6a37f3edf6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:26.822887 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4c261b-4d7d-4abb-a996-f6a37f3edf6b admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:26.823681 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4c261b-4d7d-4abb-a996-f6a37f3edf6b admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:26.824396 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4c261b-4d7d-4abb-a996-f6a37f3edf6b admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:26.825716 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba4c261b-4d7d-4abb-a996-f6a37f3edf6b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:26.829701 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba4c261b-4d7d-4abb-a996-f6a37f3edf6b admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:26.858417 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1068/2096] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:41:26 2022] POST /identity/v3/projects => generated 369 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:41:26.864799 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66776553-6e9a-464e-8817-3826214c1f94 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:26.939006 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1029/2097] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:41:26 2022] POST /identity/v3/users => generated 292 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:41:26.943868 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4c261b-4d7d-4abb-a996-f6a37f3edf6b admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:26.959700 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4c261b-4d7d-4abb-a996-f6a37f3edf6b admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:26.963776 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7545384-7def-4d9f-9d4e-f77731d86f51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:26.964470 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7545384-7def-4d9f-9d4e-f77731d86f51 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:26.964761 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7545384-7def-4d9f-9d4e-f77731d86f51 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:26.965047 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7545384-7def-4d9f-9d4e-f77731d86f51 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:26.965793 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7545384-7def-4d9f-9d4e-f77731d86f51 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:26.965894 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66776553-6e9a-464e-8817-3826214c1f94 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:26.967574 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7545384-7def-4d9f-9d4e-f77731d86f51 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:26.974189 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1c89b5-e998-4cd7-ad72-8eb834675ad7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:26.974868 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1c89b5-e998-4cd7-ad72-8eb834675ad7 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:26.975382 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1c89b5-e998-4cd7-ad72-8eb834675ad7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:26.975871 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1c89b5-e998-4cd7-ad72-8eb834675ad7 admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:26.977120 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1c89b5-e998-4cd7-ad72-8eb834675ad7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:26.982200 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1030/2098] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:26 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 12 17:41:26.982382 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1c89b5-e998-4cd7-ad72-8eb834675ad7 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:26.983355 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7545384-7def-4d9f-9d4e-f77731d86f51 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:26.998025 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7545384-7def-4d9f-9d4e-f77731d86f51 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:26.999695 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea2c31c7-97d4-4e1c-99e0-a0829efd885b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:27.001131 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2c31c7-97d4-4e1c-99e0-a0829efd885b admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:27.001378 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2c31c7-97d4-4e1c-99e0-a0829efd885b admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:27.001687 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea2c31c7-97d4-4e1c-99e0-a0829efd885b admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:27.002425 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea2c31c7-97d4-4e1c-99e0-a0829efd885b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:27.004080 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea2c31c7-97d4-4e1c-99e0-a0829efd885b admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:27.027433 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1031/2099] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:26 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:41:27.038699 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea2c31c7-97d4-4e1c-99e0-a0829efd885b admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:27.064145 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea2c31c7-97d4-4e1c-99e0-a0829efd885b admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:27.071091 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331595f5-70e2-4b43-a3d9-f5986ca572d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:27.072056 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331595f5-70e2-4b43-a3d9-f5986ca572d2 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:27.072344 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331595f5-70e2-4b43-a3d9-f5986ca572d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:27.072633 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331595f5-70e2-4b43-a3d9-f5986ca572d2 admin admin] PATH_INFO: `/v3/projects/1fcfbb917d9e48a8838957192053f7d0/users/8a279c7e8015409094f835fdee34f6fd/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:27.091940 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331595f5-70e2-4b43-a3d9-f5986ca572d2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1fcfbb917d9e48a8838957192053f7d0, user_id=8a279c7e8015409094f835fdee34f6fd, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:27.096886 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1069/2100] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:41:26 2022] POST /identity/v3/users => generated 291 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:41:27.097026 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331595f5-70e2-4b43-a3d9-f5986ca572d2 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:27.097948 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1c89b5-e998-4cd7-ad72-8eb834675ad7 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:27.108485 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1c89b5-e998-4cd7-ad72-8eb834675ad7 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:27.114209 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e88cf7a-9342-422b-8518-9704c92ee0cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:27.114941 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e88cf7a-9342-422b-8518-9704c92ee0cd admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:27.115230 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e88cf7a-9342-422b-8518-9704c92ee0cd admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:27.115599 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e88cf7a-9342-422b-8518-9704c92ee0cd admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:27.116515 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e88cf7a-9342-422b-8518-9704c92ee0cd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:27.117973 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e88cf7a-9342-422b-8518-9704c92ee0cd admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:27.133122 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1070/2101] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:27 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:41:27.134032 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e88cf7a-9342-422b-8518-9704c92ee0cd admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:27.146629 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e88cf7a-9342-422b-8518-9704c92ee0cd admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:27.151848 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-060bd229-b271-418a-af86-29b9fc119ae7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:27.152858 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060bd229-b271-418a-af86-29b9fc119ae7 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:27.153152 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060bd229-b271-418a-af86-29b9fc119ae7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:27.154249 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060bd229-b271-418a-af86-29b9fc119ae7 admin admin] PATH_INFO: `/v3/projects/94e44440d34e45efab921cc45d2c2ea1/users/ef51f4a717b2453ea4fa8bd527cf6941/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:27.174471 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-060bd229-b271-418a-af86-29b9fc119ae7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=94e44440d34e45efab921cc45d2c2ea1, user_id=ef51f4a717b2453ea4fa8bd527cf6941, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:27.178087 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-060bd229-b271-418a-af86-29b9fc119ae7 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:27.184845 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1032/2102] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:41:27 2022] PUT /identity/v3/projects/1fcfbb917d9e48a8838957192053f7d0/users/8a279c7e8015409094f835fdee34f6fd/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:27.197524 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964407a8-1056-48e8-a0f1-7302dc6015c1 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:27.198480 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964407a8-1056-48e8-a0f1-7302dc6015c1 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:27.198480 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964407a8-1056-48e8-a0f1-7302dc6015c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:27.245949 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-964407a8-1056-48e8-a0f1-7302dc6015c1 None None] MFA Rules not processed for user `8a279c7e8015409094f835fdee34f6fd`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:27.247576 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-964407a8-1056-48e8-a0f1-7302dc6015c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:27.286754 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1071/2103] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:41:27 2022] PUT /identity/v3/projects/94e44440d34e45efab921cc45d2c2ea1/users/ef51f4a717b2453ea4fa8bd527cf6941/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:27.291489 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-060bd229-b271-418a-af86-29b9fc119ae7 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:27.361955 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1033/2104] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:27 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:41:27.363073 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964407a8-1056-48e8-a0f1-7302dc6015c1 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:27.423021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-060bd229-b271-418a-af86-29b9fc119ae7 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:27.431613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be84397c-f829-4b27-a2a2-516a81e5aa81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:27.432674 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be84397c-f829-4b27-a2a2-516a81e5aa81 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:27.432971 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be84397c-f829-4b27-a2a2-516a81e5aa81 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:27.433358 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be84397c-f829-4b27-a2a2-516a81e5aa81 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:27.434683 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be84397c-f829-4b27-a2a2-516a81e5aa81 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:27.436546 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be84397c-f829-4b27-a2a2-516a81e5aa81 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:27.449920 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964407a8-1056-48e8-a0f1-7302dc6015c1 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:27.455317 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1072/2105] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:27 2022] GET /identity/v3/roles => generated 1420 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 12 17:41:27.456319 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be84397c-f829-4b27-a2a2-516a81e5aa81 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:27.465269 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95853c5-1513-4c23-b7c3-ac43c49cb3f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:27.467209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95853c5-1513-4c23-b7c3-ac43c49cb3f1 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:27.467691 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95853c5-1513-4c23-b7c3-ac43c49cb3f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:27.469522 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be84397c-f829-4b27-a2a2-516a81e5aa81 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:27.470520 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95853c5-1513-4c23-b7c3-ac43c49cb3f1 admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:27.472236 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c95853c5-1513-4c23-b7c3-ac43c49cb3f1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:27.474214 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7aa0b2b-ad43-4d90-9d69-c15b89fedbed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:27.475034 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c95853c5-1513-4c23-b7c3-ac43c49cb3f1 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:27.475297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7aa0b2b-ad43-4d90-9d69-c15b89fedbed admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:27.475588 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7aa0b2b-ad43-4d90-9d69-c15b89fedbed admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:27.475871 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7aa0b2b-ad43-4d90-9d69-c15b89fedbed admin admin] PATH_INFO: `/v3/projects/94e44440d34e45efab921cc45d2c2ea1/users/ef51f4a717b2453ea4fa8bd527cf6941/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:27.481767 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7aa0b2b-ad43-4d90-9d69-c15b89fedbed admin admin] RBAC: Authorizing `identity:create_grant(project_id=94e44440d34e45efab921cc45d2c2ea1, user_id=ef51f4a717b2453ea4fa8bd527cf6941, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:27.489271 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7aa0b2b-ad43-4d90-9d69-c15b89fedbed admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:27.497117 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:27.509537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1073/2106] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:41:27 2022] PUT /identity/v3/projects/94e44440d34e45efab921cc45d2c2ea1/users/ef51f4a717b2453ea4fa8bd527cf6941/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:27.521255 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866fac87-9972-4da0-87f0-fa558950c822 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:27.521554 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866fac87-9972-4da0-87f0-fa558950c822 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:27.521993 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866fac87-9972-4da0-87f0-fa558950c822 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:27.554660 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-866fac87-9972-4da0-87f0-fa558950c822 None None] MFA Rules not processed for user `ef51f4a717b2453ea4fa8bd527cf6941`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:27.567235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-866fac87-9972-4da0-87f0-fa558950c822 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:27.592470 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1034/2107] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:41:27 2022] POST /identity/v3/projects => generated 383 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:41:27.599139 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95853c5-1513-4c23-b7c3-ac43c49cb3f1 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:27.728791 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95853c5-1513-4c23-b7c3-ac43c49cb3f1 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:27.737604 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88dffa39-65a2-459c-8d69-d0a91cc076a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:27.738871 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88dffa39-65a2-459c-8d69-d0a91cc076a9 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:27.739543 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88dffa39-65a2-459c-8d69-d0a91cc076a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:27.740133 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88dffa39-65a2-459c-8d69-d0a91cc076a9 admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:27.741030 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88dffa39-65a2-459c-8d69-d0a91cc076a9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:27.743538 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88dffa39-65a2-459c-8d69-d0a91cc076a9 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:27.766488 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1074/2108] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:27 2022] POST /identity/v3/auth/tokens => generated 3303 bytes in 241 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:41:27.774053 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866fac87-9972-4da0-87f0-fa558950c822 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:27.775574 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-866fac87-9972-4da0-87f0-fa558950c822 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:27.866232 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1035/2109] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:41:27 2022] POST /identity/v3/users => generated 298 bytes in 264 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:41:27.867346 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88dffa39-65a2-459c-8d69-d0a91cc076a9 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:27.882542 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88dffa39-65a2-459c-8d69-d0a91cc076a9 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:27.904588 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0daec570-3cac-4508-91dd-c7323680f1da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:27.906519 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866fac87-9972-4da0-87f0-fa558950c822 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:27.914462 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0daec570-3cac-4508-91dd-c7323680f1da admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:27.914462 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0daec570-3cac-4508-91dd-c7323680f1da admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:27.914462 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0daec570-3cac-4508-91dd-c7323680f1da admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:27.914462 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0daec570-3cac-4508-91dd-c7323680f1da admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:27.914462 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0daec570-3cac-4508-91dd-c7323680f1da admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:27.937654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1036/2110] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:27 2022] GET /identity/v3/roles => generated 1420 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:41:27.943975 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0daec570-3cac-4508-91dd-c7323680f1da admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:27.957363 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0daec570-3cac-4508-91dd-c7323680f1da admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:27.972230 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e276a7-2954-465c-9cb3-70c3e5a6f378 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:27.973468 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e276a7-2954-465c-9cb3-70c3e5a6f378 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:27.974067 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e276a7-2954-465c-9cb3-70c3e5a6f378 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:27.974992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e276a7-2954-465c-9cb3-70c3e5a6f378 admin admin] PATH_INFO: `/v3/projects/95cda5cf169542fdb3ea81a15fd4bb6a/users/74f6dfd1410741f398134e4a85156a2f/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:28.021120 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e276a7-2954-465c-9cb3-70c3e5a6f378 admin admin] RBAC: Authorizing `identity:create_grant(project_id=95cda5cf169542fdb3ea81a15fd4bb6a, user_id=74f6dfd1410741f398134e4a85156a2f, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:28.027255 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e276a7-2954-465c-9cb3-70c3e5a6f378 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:28.064894 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eee6f2e-5dc3-4e92-9a06-0b726d3b5f1a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:28.066178 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eee6f2e-5dc3-4e92-9a06-0b726d3b5f1a service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:28.066783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eee6f2e-5dc3-4e92-9a06-0b726d3b5f1a service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:28.067103 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eee6f2e-5dc3-4e92-9a06-0b726d3b5f1a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:28.217262 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eee6f2e-5dc3-4e92-9a06-0b726d3b5f1a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:28.222991 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eee6f2e-5dc3-4e92-9a06-0b726d3b5f1a service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:28.236101 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1075/2111] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:41:27 2022] GET /identity/v3/auth/tokens => generated 3303 bytes in 463 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:41:28.238293 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880cdbd8-63d6-490c-ab65-765eb52b5ff7 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:28.239459 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880cdbd8-63d6-490c-ab65-765eb52b5ff7 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:28.239762 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880cdbd8-63d6-490c-ab65-765eb52b5ff7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:28.263797 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-880cdbd8-63d6-490c-ab65-765eb52b5ff7 None None] MFA Rules not processed for user `3ac0d5d8de4943a3888abee3b24b8a6f`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:28.264954 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-880cdbd8-63d6-490c-ab65-765eb52b5ff7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:28.285815 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1037/2112] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:41:27 2022] PUT /identity/v3/projects/95cda5cf169542fdb3ea81a15fd4bb6a/users/74f6dfd1410741f398134e4a85156a2f/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 339 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:28.292149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e276a7-2954-465c-9cb3-70c3e5a6f378 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:28.419591 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1076/2113] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Jul 12 17:41:28 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:28.421579 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880cdbd8-63d6-490c-ab65-765eb52b5ff7 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:28.456619 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880cdbd8-63d6-490c-ab65-765eb52b5ff7 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:28.492753 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e276a7-2954-465c-9cb3-70c3e5a6f378 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:28.510286 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5de6a1-7d6e-472a-851f-309d37411e6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:28.516078 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5de6a1-7d6e-472a-851f-309d37411e6f admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:28.516532 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5de6a1-7d6e-472a-851f-309d37411e6f admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:28.516888 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5de6a1-7d6e-472a-851f-309d37411e6f admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:28.517933 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a5de6a1-7d6e-472a-851f-309d37411e6f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:28.519841 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a5de6a1-7d6e-472a-851f-309d37411e6f admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:28.538028 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1038/2114] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:28 2022] GET /identity/v3/roles => generated 1420 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 12 17:41:28.538645 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5de6a1-7d6e-472a-851f-309d37411e6f admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:28.540880 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-4a5de6a1-7d6e-472a-851f-309d37411e6f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:28.614168 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e92789-5de3-4818-81b6-c332a6667379 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:28.616738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e92789-5de3-4818-81b6-c332a6667379 service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:28.617255 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e92789-5de3-4818-81b6-c332a6667379 service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:28.620583 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e92789-5de3-4818-81b6-c332a6667379 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:28.738474 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5de6a1-7d6e-472a-851f-309d37411e6f admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:28.756184 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-627074be-a646-48cc-9dda-d08747e9d6e1 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7906d1a19c642bd93507a27bf3e3dae', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:28.757348 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627074be-a646-48cc-9dda-d08747e9d6e1 service swift] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:28.757741 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627074be-a646-48cc-9dda-d08747e9d6e1 service swift] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:28.762222 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627074be-a646-48cc-9dda-d08747e9d6e1 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:28.782214 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e92789-5de3-4818-81b6-c332a6667379 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:28.784141 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e92789-5de3-4818-81b6-c332a6667379 service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:28.812072 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1077/2115] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:41:28 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 392 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:41:28.816665 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e92789-5de3-4818-81b6-c332a6667379 service cinder] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:28.833852 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:28.841322 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e92789-5de3-4818-81b6-c332a6667379 service cinder] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:28.851852 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d648e659-f35d-4b2a-9c5f-b7cba2331679 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:28.853192 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d648e659-f35d-4b2a-9c5f-b7cba2331679 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:28.853595 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d648e659-f35d-4b2a-9c5f-b7cba2331679 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:28.854019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d648e659-f35d-4b2a-9c5f-b7cba2331679 admin admin] PATH_INFO: `/v3/projects/95cda5cf169542fdb3ea81a15fd4bb6a/users/74f6dfd1410741f398134e4a85156a2f/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:28.857749 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d648e659-f35d-4b2a-9c5f-b7cba2331679 admin admin] RBAC: Authorizing `identity:create_grant(project_id=95cda5cf169542fdb3ea81a15fd4bb6a, user_id=74f6dfd1410741f398134e4a85156a2f, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:28.864396 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d648e659-f35d-4b2a-9c5f-b7cba2331679 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:28.894378 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627074be-a646-48cc-9dda-d08747e9d6e1 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:28.896124 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1078/2116] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:41:28 2022] PUT /identity/v3/projects/95cda5cf169542fdb3ea81a15fd4bb6a/users/74f6dfd1410741f398134e4a85156a2f/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:28.899846 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2260709-cbc8-476b-a6c0-8dba39fe5186 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:28.901487 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2260709-cbc8-476b-a6c0-8dba39fe5186 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:28.902308 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2260709-cbc8-476b-a6c0-8dba39fe5186 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:28.914617 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-627074be-a646-48cc-9dda-d08747e9d6e1 service swift] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:28.928979 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1039/2117] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Jul 12 17:41:28 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 12 17:41:28.943905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078e7e45-0084-468d-ae8b-df82a62b8b31 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:28.943905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078e7e45-0084-468d-ae8b-df82a62b8b31 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:28.943905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078e7e45-0084-468d-ae8b-df82a62b8b31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:28.944887 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-a2260709-cbc8-476b-a6c0-8dba39fe5186 None None] MFA Rules not processed for user `c8441a4a72154ee3bbb95af66abf2ffd`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:28.944887 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-a2260709-cbc8-476b-a6c0-8dba39fe5186 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:29.001453 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-078e7e45-0084-468d-ae8b-df82a62b8b31 None None] MFA Rules not processed for user `74f6dfd1410741f398134e4a85156a2f`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:29.003454 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-078e7e45-0084-468d-ae8b-df82a62b8b31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:29.058929 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1079/2118] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:28 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:29.166323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1040/2119] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:28 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:29.491094 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:29.753471 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2260709-cbc8-476b-a6c0-8dba39fe5186 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:29.755613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-a2260709-cbc8-476b-a6c0-8dba39fe5186 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:29.847603 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2260709-cbc8-476b-a6c0-8dba39fe5186 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:29.853576 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0442f55-8406-4de5-befb-2e3983bbdda1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:29.854938 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0442f55-8406-4de5-befb-2e3983bbdda1 service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:29.855417 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0442f55-8406-4de5-befb-2e3983bbdda1 service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:29.855860 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0442f55-8406-4de5-befb-2e3983bbdda1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:29.873636 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0442f55-8406-4de5-befb-2e3983bbdda1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:29.893279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0442f55-8406-4de5-befb-2e3983bbdda1 service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:29.965706 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1080/2120] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:41:29 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 213 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:41:30.442725 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078e7e45-0084-468d-ae8b-df82a62b8b31 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:30.454462 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078e7e45-0084-468d-ae8b-df82a62b8b31 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:30.463988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2634d62-0dce-455d-81fe-4537950da2f9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:30.465045 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2634d62-0dce-455d-81fe-4537950da2f9 service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:30.465691 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2634d62-0dce-455d-81fe-4537950da2f9 service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:30.466170 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2634d62-0dce-455d-81fe-4537950da2f9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:30.540280 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2634d62-0dce-455d-81fe-4537950da2f9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:30.542068 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2634d62-0dce-455d-81fe-4537950da2f9 service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:30.567003 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1041/2121] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:41:30 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:41:30.724899 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:INSERT=5,keystone:SELECT=95 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:31.864157 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:33.010723 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:33.878219 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:34.611788 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0442f55-8406-4de5-befb-2e3983bbdda1 service cinder] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:34.683503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:34.743317 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0442f55-8406-4de5-befb-2e3983bbdda1 service cinder] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:34.750250 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14e59c6-71c0-42de-a048-4d9dc89482ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:34.755045 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14e59c6-71c0-42de-a048-4d9dc89482ea admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:34.757351 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14e59c6-71c0-42de-a048-4d9dc89482ea admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:34.757811 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14e59c6-71c0-42de-a048-4d9dc89482ea admin admin] PATH_INFO: `/v3/users/8a279c7e8015409094f835fdee34f6fd` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:34.759928 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14e59c6-71c0-42de-a048-4d9dc89482ea admin admin] RBAC: Authorizing `identity:delete_user(user_id=8a279c7e8015409094f835fdee34f6fd)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:34.764532 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14e59c6-71c0-42de-a048-4d9dc89482ea admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:35.117085 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-d14e59c6-71c0-42de-a048-4d9dc89482ea admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8a279c7e8015409094f835fdee34f6fd'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:35.117974 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-d14e59c6-71c0-42de-a048-4d9dc89482ea admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8a279c7e8015409094f835fdee34f6fd'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:35.138813 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-d14e59c6-71c0-42de-a048-4d9dc89482ea admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8a279c7e8015409094f835fdee34f6fd'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:35.166396 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-d14e59c6-71c0-42de-a048-4d9dc89482ea admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8a279c7e8015409094f835fdee34f6fd'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:35.182228 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-d14e59c6-71c0-42de-a048-4d9dc89482ea admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8a279c7e8015409094f835fdee34f6fd'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:35.305708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1081/2122] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:41:34 2022] DELETE /identity/v3/users/8a279c7e8015409094f835fdee34f6fd => generated 0 bytes in 694 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:35.310378 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed30e29e-db5e-49ac-acc4-52ef88af008a None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:35.311112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed30e29e-db5e-49ac-acc4-52ef88af008a None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:35.311112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed30e29e-db5e-49ac-acc4-52ef88af008a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:35.346107 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-ed30e29e-db5e-49ac-acc4-52ef88af008a None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:35.347344 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-ed30e29e-db5e-49ac-acc4-52ef88af008a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:35.615723 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1042/2123] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 306 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:35.631247 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14e59c6-71c0-42de-a048-4d9dc89482ea admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:35.633419 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-d14e59c6-71c0-42de-a048-4d9dc89482ea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:35.787961 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14e59c6-71c0-42de-a048-4d9dc89482ea admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:35.797157 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9409c4-729a-4502-a753-193912061908 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:35.798244 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9409c4-729a-4502-a753-193912061908 service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:35.799410 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9409c4-729a-4502-a753-193912061908 service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:35.799778 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9409c4-729a-4502-a753-193912061908 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:35.826924 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9409c4-729a-4502-a753-193912061908 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:35.832134 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9409c4-729a-4502-a753-193912061908 service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:35.862711 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1082/2124] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:41:35 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 232 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:41:36.789733 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed30e29e-db5e-49ac-acc4-52ef88af008a None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:36.795225 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-ed30e29e-db5e-49ac-acc4-52ef88af008a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:36.830144 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed30e29e-db5e-49ac-acc4-52ef88af008a None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:36.830408 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9409c4-729a-4502-a753-193912061908 service neutron] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:36.831709 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-bc9409c4-729a-4502-a753-193912061908 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:36.839521 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e26d94-2142-4d0c-9518-cc4a6da27538 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:36.840789 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e26d94-2142-4d0c-9518-cc4a6da27538 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:36.841307 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e26d94-2142-4d0c-9518-cc4a6da27538 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:36.846457 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e26d94-2142-4d0c-9518-cc4a6da27538 admin admin] PATH_INFO: `/v3/users/1dc3556822484d24967f2288813c8bee` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:36.846457 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e26d94-2142-4d0c-9518-cc4a6da27538 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1dc3556822484d24967f2288813c8bee)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:36.847507 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e26d94-2142-4d0c-9518-cc4a6da27538 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:36.859880 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9409c4-729a-4502-a753-193912061908 service neutron] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:36.874731 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e27c39b-41b9-427a-b32f-7a9a8ccd8d42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:36.876581 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e27c39b-41b9-427a-b32f-7a9a8ccd8d42 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:36.877078 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e27c39b-41b9-427a-b32f-7a9a8ccd8d42 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:36.877540 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e27c39b-41b9-427a-b32f-7a9a8ccd8d42 admin admin] PATH_INFO: `/v3/projects/1fcfbb917d9e48a8838957192053f7d0` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:36.882134 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e27c39b-41b9-427a-b32f-7a9a8ccd8d42 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1fcfbb917d9e48a8838957192053f7d0)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:36.884498 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e27c39b-41b9-427a-b32f-7a9a8ccd8d42 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:37.146071 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:37.165438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-5e27c39b-41b9-427a-b32f-7a9a8ccd8d42 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1fcfbb917d9e48a8838957192053f7d0', 'request_id': 'req-5e27c39b-41b9-427a-b32f-7a9a8ccd8d42'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:37.184680 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-5e27c39b-41b9-427a-b32f-7a9a8ccd8d42 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1fcfbb917d9e48a8838957192053f7d0', 'request_id': 'req-5e27c39b-41b9-427a-b32f-7a9a8ccd8d42'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:37.202905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-b4e26d94-2142-4d0c-9518-cc4a6da27538 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1dc3556822484d24967f2288813c8bee'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:37.335532 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-5e27c39b-41b9-427a-b32f-7a9a8ccd8d42 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1fcfbb917d9e48a8838957192053f7d0', 'request_id': 'req-5e27c39b-41b9-427a-b32f-7a9a8ccd8d42'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:37.401416 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-5e27c39b-41b9-427a-b32f-7a9a8ccd8d42 admin admin] The token cache is being invalidate because project 1fcfbb917d9e48a8838957192053f7d0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:41:37.406386 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-5e27c39b-41b9-427a-b32f-7a9a8ccd8d42 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:37.406386 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-5e27c39b-41b9-427a-b32f-7a9a8ccd8d42 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:37.413188 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1083/2125] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:41:36 2022] DELETE /identity/v3/projects/1fcfbb917d9e48a8838957192053f7d0 => generated 0 bytes in 583 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:41:37.424999 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e27c39b-41b9-427a-b32f-7a9a8ccd8d42 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:37.427462 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-b4e26d94-2142-4d0c-9518-cc4a6da27538 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1dc3556822484d24967f2288813c8bee'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:37.428138 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-b4e26d94-2142-4d0c-9518-cc4a6da27538 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1dc3556822484d24967f2288813c8bee'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:37.435056 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-5e27c39b-41b9-427a-b32f-7a9a8ccd8d42 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:37.475583 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-b4e26d94-2142-4d0c-9518-cc4a6da27538 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1dc3556822484d24967f2288813c8bee'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:37.556181 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-b4e26d94-2142-4d0c-9518-cc4a6da27538 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1dc3556822484d24967f2288813c8bee'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:37.604908 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1043/2126] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:41:36 2022] DELETE /identity/v3/users/1dc3556822484d24967f2288813c8bee => generated 0 bytes in 816 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:37.614876 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef290369-e857-400d-a526-18efa8bef4b6 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:37.618235 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef290369-e857-400d-a526-18efa8bef4b6 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:37.619415 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef290369-e857-400d-a526-18efa8bef4b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:37.684959 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-ef290369-e857-400d-a526-18efa8bef4b6 None None] MFA Rules not processed for user `ce09df98c929416e8ca67b0728349bfb`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:37.687068 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-ef290369-e857-400d-a526-18efa8bef4b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:37.817558 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:37.828248 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1044/2127] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:37 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:37.831025 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cf2bd6-44f4-48a3-9b4d-55eb74cf57f5 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:37.831604 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cf2bd6-44f4-48a3-9b4d-55eb74cf57f5 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:37.832077 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cf2bd6-44f4-48a3-9b4d-55eb74cf57f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:37.861857 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-33cf2bd6-44f4-48a3-9b4d-55eb74cf57f5 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:37.863499 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-33cf2bd6-44f4-48a3-9b4d-55eb74cf57f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:37.868362 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e27c39b-41b9-427a-b32f-7a9a8ccd8d42 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:38.103073 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1045/2128] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 273 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:38.111164 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a164017-e2e7-43a0-bfba-f004a9da7cfd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:38.112588 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a164017-e2e7-43a0-bfba-f004a9da7cfd admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:38.113421 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a164017-e2e7-43a0-bfba-f004a9da7cfd admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:38.114282 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a164017-e2e7-43a0-bfba-f004a9da7cfd admin admin] PATH_INFO: `/v3/users/74f6dfd1410741f398134e4a85156a2f` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:38.120071 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a164017-e2e7-43a0-bfba-f004a9da7cfd admin admin] RBAC: Authorizing `identity:delete_user(user_id=74f6dfd1410741f398134e4a85156a2f)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:38.126238 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a164017-e2e7-43a0-bfba-f004a9da7cfd admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:38.149518 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33cf2bd6-44f4-48a3-9b4d-55eb74cf57f5 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:38.153536 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-33cf2bd6-44f4-48a3-9b4d-55eb74cf57f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:38.248513 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33cf2bd6-44f4-48a3-9b4d-55eb74cf57f5 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:38.254103 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6376bec5-1b2f-43e1-9390-3fe718169d6e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:38.255263 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6376bec5-1b2f-43e1-9390-3fe718169d6e service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:38.256075 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6376bec5-1b2f-43e1-9390-3fe718169d6e service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:38.256736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6376bec5-1b2f-43e1-9390-3fe718169d6e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:38.273264 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6376bec5-1b2f-43e1-9390-3fe718169d6e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:38.279555 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6376bec5-1b2f-43e1-9390-3fe718169d6e service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:38.291213 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1046/2129] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:41:38 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:41:38.435414 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-0a164017-e2e7-43a0-bfba-f004a9da7cfd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '74f6dfd1410741f398134e4a85156a2f'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:38.437048 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-0a164017-e2e7-43a0-bfba-f004a9da7cfd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '74f6dfd1410741f398134e4a85156a2f'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:38.439078 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-0a164017-e2e7-43a0-bfba-f004a9da7cfd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '74f6dfd1410741f398134e4a85156a2f'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:38.463131 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-0a164017-e2e7-43a0-bfba-f004a9da7cfd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '74f6dfd1410741f398134e4a85156a2f'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:38.503288 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-0a164017-e2e7-43a0-bfba-f004a9da7cfd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '74f6dfd1410741f398134e4a85156a2f'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:38.509218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:38.597109 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1084/2130] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:41:37 2022] DELETE /identity/v3/users/74f6dfd1410741f398134e4a85156a2f => generated 0 bytes in 1173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:38.766245 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6376bec5-1b2f-43e1-9390-3fe718169d6e service neutron] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:38.768203 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-6376bec5-1b2f-43e1-9390-3fe718169d6e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:38.799046 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6376bec5-1b2f-43e1-9390-3fe718169d6e service neutron] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:38.993729 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f938586e-a0c4-4928-b2ca-1ae75a3c5073 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:38.994791 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f938586e-a0c4-4928-b2ca-1ae75a3c5073 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:38.994791 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f938586e-a0c4-4928-b2ca-1ae75a3c5073 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:38.995140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f938586e-a0c4-4928-b2ca-1ae75a3c5073 admin admin] PATH_INFO: `/v3/users/44ff65795bf646429b317820c8af4e3c` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:38.996960 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f938586e-a0c4-4928-b2ca-1ae75a3c5073 admin admin] RBAC: Authorizing `identity:delete_user(user_id=44ff65795bf646429b317820c8af4e3c)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:39.001480 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f938586e-a0c4-4928-b2ca-1ae75a3c5073 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:39.336936 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:39.344718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-f938586e-a0c4-4928-b2ca-1ae75a3c5073 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '44ff65795bf646429b317820c8af4e3c'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:39.485294 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-f938586e-a0c4-4928-b2ca-1ae75a3c5073 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '44ff65795bf646429b317820c8af4e3c'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:39.486293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-f938586e-a0c4-4928-b2ca-1ae75a3c5073 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '44ff65795bf646429b317820c8af4e3c'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:39.501371 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-f938586e-a0c4-4928-b2ca-1ae75a3c5073 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '44ff65795bf646429b317820c8af4e3c'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:39.527579 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-f938586e-a0c4-4928-b2ca-1ae75a3c5073 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '44ff65795bf646429b317820c8af4e3c'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:39.544699 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1047/2131] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:41:38 2022] DELETE /identity/v3/users/44ff65795bf646429b317820c8af4e3c => generated 0 bytes in 775 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:39.546511 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32045c17-013c-4365-ae97-1b4e4dddd0cf None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:39.547659 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32045c17-013c-4365-ae97-1b4e4dddd0cf None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:39.549263 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32045c17-013c-4365-ae97-1b4e4dddd0cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:39.589470 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-32045c17-013c-4365-ae97-1b4e4dddd0cf None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:39.591348 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-32045c17-013c-4365-ae97-1b4e4dddd0cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:39.676467 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f938586e-a0c4-4928-b2ca-1ae75a3c5073 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:39.678946 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-f938586e-a0c4-4928-b2ca-1ae75a3c5073 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:39.723783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1085/2132] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:39.725973 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32045c17-013c-4365-ae97-1b4e4dddd0cf None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:39.732313 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-32045c17-013c-4365-ae97-1b4e4dddd0cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:39.851379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f938586e-a0c4-4928-b2ca-1ae75a3c5073 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:39.858638 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963ca036-d79b-4115-ba86-ca050a60450a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:39.859968 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963ca036-d79b-4115-ba86-ca050a60450a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:39.860560 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963ca036-d79b-4115-ba86-ca050a60450a admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:39.867835 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963ca036-d79b-4115-ba86-ca050a60450a admin admin] PATH_INFO: `/v3/projects/884c4406f6de4c4fbb56866fdad40575` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:39.869650 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963ca036-d79b-4115-ba86-ca050a60450a admin admin] RBAC: Authorizing `identity:delete_project(project_id=884c4406f6de4c4fbb56866fdad40575)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:39.872745 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963ca036-d79b-4115-ba86-ca050a60450a admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:39.873084 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32045c17-013c-4365-ae97-1b4e4dddd0cf None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:39.882994 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a129df-29d4-489b-89e8-8dc3a2a5b8ba service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:39.884210 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a129df-29d4-489b-89e8-8dc3a2a5b8ba service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:39.885579 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a129df-29d4-489b-89e8-8dc3a2a5b8ba service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:39.885740 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a129df-29d4-489b-89e8-8dc3a2a5b8ba service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:39.902751 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a129df-29d4-489b-89e8-8dc3a2a5b8ba service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:39.904534 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a129df-29d4-489b-89e8-8dc3a2a5b8ba service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:39.920302 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1086/2133] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:41:39 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 196 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:41:39.921645 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a129df-29d4-489b-89e8-8dc3a2a5b8ba service neutron] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:39.924233 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-23a129df-29d4-489b-89e8-8dc3a2a5b8ba service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:39.945966 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a129df-29d4-489b-89e8-8dc3a2a5b8ba service neutron] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:39.952975 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b670f5c-a1e7-4d83-b4f1-68a629263b86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:39.954262 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b670f5c-a1e7-4d83-b4f1-68a629263b86 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:39.954580 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b670f5c-a1e7-4d83-b4f1-68a629263b86 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:39.955015 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b670f5c-a1e7-4d83-b4f1-68a629263b86 admin admin] PATH_INFO: `/v3/projects/95cda5cf169542fdb3ea81a15fd4bb6a` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:39.957068 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b670f5c-a1e7-4d83-b4f1-68a629263b86 admin admin] RBAC: Authorizing `identity:delete_project(project_id=95cda5cf169542fdb3ea81a15fd4bb6a)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:39.962511 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b670f5c-a1e7-4d83-b4f1-68a629263b86 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:40.050188 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-963ca036-d79b-4115-ba86-ca050a60450a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '884c4406f6de4c4fbb56866fdad40575', 'request_id': 'req-963ca036-d79b-4115-ba86-ca050a60450a'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:40.066343 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-963ca036-d79b-4115-ba86-ca050a60450a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '884c4406f6de4c4fbb56866fdad40575', 'request_id': 'req-963ca036-d79b-4115-ba86-ca050a60450a'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:40.080802 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-963ca036-d79b-4115-ba86-ca050a60450a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '884c4406f6de4c4fbb56866fdad40575', 'request_id': 'req-963ca036-d79b-4115-ba86-ca050a60450a'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:40.152580 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-963ca036-d79b-4115-ba86-ca050a60450a admin admin] The token cache is being invalidate because project 884c4406f6de4c4fbb56866fdad40575 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:41:40.152952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-963ca036-d79b-4115-ba86-ca050a60450a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:40.154322 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-963ca036-d79b-4115-ba86-ca050a60450a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:40.164598 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1048/2134] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:41:39 2022] DELETE /identity/v3/projects/884c4406f6de4c4fbb56866fdad40575 => generated 0 bytes in 481 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:41:40.166395 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963ca036-d79b-4115-ba86-ca050a60450a admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:40.170246 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-963ca036-d79b-4115-ba86-ca050a60450a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:40.216320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-6b670f5c-a1e7-4d83-b4f1-68a629263b86 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '95cda5cf169542fdb3ea81a15fd4bb6a', 'request_id': 'req-6b670f5c-a1e7-4d83-b4f1-68a629263b86'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:40.242077 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-6b670f5c-a1e7-4d83-b4f1-68a629263b86 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '95cda5cf169542fdb3ea81a15fd4bb6a', 'request_id': 'req-6b670f5c-a1e7-4d83-b4f1-68a629263b86'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:40.305694 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:40.322894 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963ca036-d79b-4115-ba86-ca050a60450a admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:40.348928 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-6b670f5c-a1e7-4d83-b4f1-68a629263b86 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '95cda5cf169542fdb3ea81a15fd4bb6a', 'request_id': 'req-6b670f5c-a1e7-4d83-b4f1-68a629263b86'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:40.398715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-6b670f5c-a1e7-4d83-b4f1-68a629263b86 admin admin] The token cache is being invalidate because project 95cda5cf169542fdb3ea81a15fd4bb6a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:41:40.399107 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-6b670f5c-a1e7-4d83-b4f1-68a629263b86 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:40.400099 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-6b670f5c-a1e7-4d83-b4f1-68a629263b86 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:40.402493 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1087/2135] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:41:39 2022] DELETE /identity/v3/projects/95cda5cf169542fdb3ea81a15fd4bb6a => generated 0 bytes in 481 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:40.411095 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741648c1-6192-418d-b519-97b59a4a9106 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:40.415513 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741648c1-6192-418d-b519-97b59a4a9106 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:40.415513 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741648c1-6192-418d-b519-97b59a4a9106 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:40.435487 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89956245-41ba-4bff-bb04-2f72b1020858 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:40.436424 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89956245-41ba-4bff-bb04-2f72b1020858 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:40.436714 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89956245-41ba-4bff-bb04-2f72b1020858 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:40.436997 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89956245-41ba-4bff-bb04-2f72b1020858 admin admin] PATH_INFO: `/v3/users/ef51f4a717b2453ea4fa8bd527cf6941` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:40.438269 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89956245-41ba-4bff-bb04-2f72b1020858 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ef51f4a717b2453ea4fa8bd527cf6941)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:40.443994 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89956245-41ba-4bff-bb04-2f72b1020858 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:40.462705 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-741648c1-6192-418d-b519-97b59a4a9106 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:40.464695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-741648c1-6192-418d-b519-97b59a4a9106 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:40.488496 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1088/2136] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:40.496958 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2709cd-3fae-453d-9d8f-964faab37152 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:40.497653 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2709cd-3fae-453d-9d8f-964faab37152 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:40.497653 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2709cd-3fae-453d-9d8f-964faab37152 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:40.544191 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-7a2709cd-3fae-453d-9d8f-964faab37152 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:40.546325 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-7a2709cd-3fae-453d-9d8f-964faab37152 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:40.574716 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1089/2137] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:41:40.575708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2709cd-3fae-453d-9d8f-964faab37152 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:40.579960 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-7a2709cd-3fae-453d-9d8f-964faab37152 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:40.608486 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2709cd-3fae-453d-9d8f-964faab37152 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:40.615615 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97525059-ba2d-46a1-8cdc-cadc2132da2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:40.616529 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97525059-ba2d-46a1-8cdc-cadc2132da2f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:40.616839 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97525059-ba2d-46a1-8cdc-cadc2132da2f admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:40.617102 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97525059-ba2d-46a1-8cdc-cadc2132da2f admin admin] PATH_INFO: `/v3/projects/f71cf15bea67452a9ce256e788bb3bfe` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:40.618507 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97525059-ba2d-46a1-8cdc-cadc2132da2f admin admin] RBAC: Authorizing `identity:delete_project(project_id=f71cf15bea67452a9ce256e788bb3bfe)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:40.622991 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97525059-ba2d-46a1-8cdc-cadc2132da2f admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:40.627454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:DELETE=8,keystone:SELECT=91,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:40.982561 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-89956245-41ba-4bff-bb04-2f72b1020858 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ef51f4a717b2453ea4fa8bd527cf6941'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:41.007932 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-97525059-ba2d-46a1-8cdc-cadc2132da2f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f71cf15bea67452a9ce256e788bb3bfe', 'request_id': 'req-97525059-ba2d-46a1-8cdc-cadc2132da2f'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:41.036936 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-97525059-ba2d-46a1-8cdc-cadc2132da2f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f71cf15bea67452a9ce256e788bb3bfe', 'request_id': 'req-97525059-ba2d-46a1-8cdc-cadc2132da2f'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:41.119531 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-89956245-41ba-4bff-bb04-2f72b1020858 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ef51f4a717b2453ea4fa8bd527cf6941'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:41.121023 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-89956245-41ba-4bff-bb04-2f72b1020858 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ef51f4a717b2453ea4fa8bd527cf6941'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:41.135029 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-89956245-41ba-4bff-bb04-2f72b1020858 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ef51f4a717b2453ea4fa8bd527cf6941'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:41.165687 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-89956245-41ba-4bff-bb04-2f72b1020858 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ef51f4a717b2453ea4fa8bd527cf6941'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:41.181854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1049/2138] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:41:40 2022] DELETE /identity/v3/users/ef51f4a717b2453ea4fa8bd527cf6941 => generated 0 bytes in 1015 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:41.186175 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89956245-41ba-4bff-bb04-2f72b1020858 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:41.186175 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-89956245-41ba-4bff-bb04-2f72b1020858 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:41.204548 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-97525059-ba2d-46a1-8cdc-cadc2132da2f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f71cf15bea67452a9ce256e788bb3bfe', 'request_id': 'req-97525059-ba2d-46a1-8cdc-cadc2132da2f'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:41.238811 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-97525059-ba2d-46a1-8cdc-cadc2132da2f admin admin] The token cache is being invalidate because project f71cf15bea67452a9ce256e788bb3bfe was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:41:41.238811 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-97525059-ba2d-46a1-8cdc-cadc2132da2f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:41.238811 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-97525059-ba2d-46a1-8cdc-cadc2132da2f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:41.242903 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1090/2139] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:41:40 2022] DELETE /identity/v3/projects/f71cf15bea67452a9ce256e788bb3bfe => generated 0 bytes in 666 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:41.360035 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89956245-41ba-4bff-bb04-2f72b1020858 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:41.373897 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8932579f-4157-4650-82b5-2d0271fcc17b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:41.376793 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8932579f-4157-4650-82b5-2d0271fcc17b admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:41.377539 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8932579f-4157-4650-82b5-2d0271fcc17b admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:41.378205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8932579f-4157-4650-82b5-2d0271fcc17b admin admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:41.382053 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8932579f-4157-4650-82b5-2d0271fcc17b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:41:41.393777 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8932579f-4157-4650-82b5-2d0271fcc17b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:41.398981 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8932579f-4157-4650-82b5-2d0271fcc17b admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:41.433113 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1050/2140] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:41:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:41:41.445715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97525059-ba2d-46a1-8cdc-cadc2132da2f admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:41.449583 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-97525059-ba2d-46a1-8cdc-cadc2132da2f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:41.595375 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97525059-ba2d-46a1-8cdc-cadc2132da2f admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:41.606861 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759dc065-5b46-4871-9e6b-b78d23a419d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:41.608329 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759dc065-5b46-4871-9e6b-b78d23a419d9 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:41.608856 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759dc065-5b46-4871-9e6b-b78d23a419d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:41.610736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759dc065-5b46-4871-9e6b-b78d23a419d9 admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:41.614482 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759dc065-5b46-4871-9e6b-b78d23a419d9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:41.618593 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759dc065-5b46-4871-9e6b-b78d23a419d9 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:41.749927 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1091/2141] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:41:41 2022] POST /identity/v3/projects => generated 381 bytes in 308 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:41:41.751546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8932579f-4157-4650-82b5-2d0271fcc17b admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:41.796183 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8932579f-4157-4650-82b5-2d0271fcc17b admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:41.842168 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:41.994345 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dcb1a8e-d6a3-426b-8c9c-288c2b88e971 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:41.995514 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dcb1a8e-d6a3-426b-8c9c-288c2b88e971 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:41.995906 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dcb1a8e-d6a3-426b-8c9c-288c2b88e971 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:41.996466 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dcb1a8e-d6a3-426b-8c9c-288c2b88e971 admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:41.998090 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dcb1a8e-d6a3-426b-8c9c-288c2b88e971 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:42.002110 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dcb1a8e-d6a3-426b-8c9c-288c2b88e971 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:42.076653 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759dc065-5b46-4871-9e6b-b78d23a419d9 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:42.079703 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-759dc065-5b46-4871-9e6b-b78d23a419d9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:42.174874 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759dc065-5b46-4871-9e6b-b78d23a419d9 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:42.176770 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1051/2142] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:41:41 2022] POST /identity/v3/users => generated 291 bytes in 426 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:41:42.189446 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dcb1a8e-d6a3-426b-8c9c-288c2b88e971 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:42.211597 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8fcbe0-418e-4c79-b3c5-35fec39df6ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:42.217545 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8fcbe0-418e-4c79-b3c5-35fec39df6ee admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:42.218303 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8fcbe0-418e-4c79-b3c5-35fec39df6ee admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:42.224195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8fcbe0-418e-4c79-b3c5-35fec39df6ee admin admin] PATH_INFO: `/v3/projects/94e44440d34e45efab921cc45d2c2ea1` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:42.230329 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8fcbe0-418e-4c79-b3c5-35fec39df6ee admin admin] RBAC: Authorizing `identity:delete_project(project_id=94e44440d34e45efab921cc45d2c2ea1)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:42.237003 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dcb1a8e-d6a3-426b-8c9c-288c2b88e971 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:42.241750 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8fcbe0-418e-4c79-b3c5-35fec39df6ee admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:42.260998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa38376c-2c12-4966-afe3-dd4e7e8eb4f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:42.261727 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa38376c-2c12-4966-afe3-dd4e7e8eb4f2 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:42.268145 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa38376c-2c12-4966-afe3-dd4e7e8eb4f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:42.269438 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa38376c-2c12-4966-afe3-dd4e7e8eb4f2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:42.275342 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa38376c-2c12-4966-afe3-dd4e7e8eb4f2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:42.281276 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa38376c-2c12-4966-afe3-dd4e7e8eb4f2 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:42.331729 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1052/2143] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:42 2022] GET /identity/v3/roles => generated 1420 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:41:42.333986 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa38376c-2c12-4966-afe3-dd4e7e8eb4f2 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:42.357107 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa38376c-2c12-4966-afe3-dd4e7e8eb4f2 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:42.367649 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f976e0e6-b800-4551-b36e-b8015e5ab854 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:42.369266 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f976e0e6-b800-4551-b36e-b8015e5ab854 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:42.370073 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f976e0e6-b800-4551-b36e-b8015e5ab854 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:42.371325 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f976e0e6-b800-4551-b36e-b8015e5ab854 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:42.373361 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f976e0e6-b800-4551-b36e-b8015e5ab854 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:42.377735 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f976e0e6-b800-4551-b36e-b8015e5ab854 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:42.409925 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1053/2144] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:42 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:41:42.415568 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f976e0e6-b800-4551-b36e-b8015e5ab854 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:42.439824 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f976e0e6-b800-4551-b36e-b8015e5ab854 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:42.449916 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7084573-3a6b-4811-bc99-e4e49357b32c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:42.451681 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7084573-3a6b-4811-bc99-e4e49357b32c admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:42.452569 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7084573-3a6b-4811-bc99-e4e49357b32c admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:42.453007 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7084573-3a6b-4811-bc99-e4e49357b32c admin admin] PATH_INFO: `/v3/projects/11a327f750534f3b8575a5f17bde34be/users/cfd0ef55a4fe45559b3b8834ad8d5557/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:42.492147 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7084573-3a6b-4811-bc99-e4e49357b32c admin admin] RBAC: Authorizing `identity:create_grant(project_id=11a327f750534f3b8575a5f17bde34be, user_id=cfd0ef55a4fe45559b3b8834ad8d5557, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:42.496820 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7084573-3a6b-4811-bc99-e4e49357b32c admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:42.505026 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ff8fcbe0-418e-4c79-b3c5-35fec39df6ee admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '94e44440d34e45efab921cc45d2c2ea1', 'request_id': 'req-ff8fcbe0-418e-4c79-b3c5-35fec39df6ee'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:42.516853 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ff8fcbe0-418e-4c79-b3c5-35fec39df6ee admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '94e44440d34e45efab921cc45d2c2ea1', 'request_id': 'req-ff8fcbe0-418e-4c79-b3c5-35fec39df6ee'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:42.605857 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1054/2145] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:41:42 2022] PUT /identity/v3/projects/11a327f750534f3b8575a5f17bde34be/users/cfd0ef55a4fe45559b3b8834ad8d5557/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:42.627591 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ff8fcbe0-418e-4c79-b3c5-35fec39df6ee admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '94e44440d34e45efab921cc45d2c2ea1', 'request_id': 'req-ff8fcbe0-418e-4c79-b3c5-35fec39df6ee'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:42.628290 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686b66d3-82df-44d9-8711-c487f910842c None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:42.629034 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686b66d3-82df-44d9-8711-c487f910842c None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:42.629513 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686b66d3-82df-44d9-8711-c487f910842c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:42.639357 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:42.660882 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ff8fcbe0-418e-4c79-b3c5-35fec39df6ee admin admin] The token cache is being invalidate because project 94e44440d34e45efab921cc45d2c2ea1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:41:42.662892 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ff8fcbe0-418e-4c79-b3c5-35fec39df6ee admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:42.665835 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ff8fcbe0-418e-4c79-b3c5-35fec39df6ee admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:42.670414 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1092/2146] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:41:42 2022] DELETE /identity/v3/projects/94e44440d34e45efab921cc45d2c2ea1 => generated 0 bytes in 594 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:42.676823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3477f225-779c-4de1-b47f-14c28c3c5ac8 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:42.677372 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3477f225-779c-4de1-b47f-14c28c3c5ac8 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:42.682516 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3477f225-779c-4de1-b47f-14c28c3c5ac8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:42.688107 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-686b66d3-82df-44d9-8711-c487f910842c None None] MFA Rules not processed for user `cfd0ef55a4fe45559b3b8834ad8d5557`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:42.691716 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-686b66d3-82df-44d9-8711-c487f910842c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:42.710417 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-3477f225-779c-4de1-b47f-14c28c3c5ac8 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:42.727260 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-3477f225-779c-4de1-b47f-14c28c3c5ac8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:42.885149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1055/2147] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:42 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 257 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:42.885149 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-686b66d3-82df-44d9-8711-c487f910842c None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:42.889675 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-686b66d3-82df-44d9-8711-c487f910842c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:43.015077 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1093/2148] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 340 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:43.023628 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305930d3-73bf-48d8-9f6f-584ffe5b234a None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:43.023944 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305930d3-73bf-48d8-9f6f-584ffe5b234a None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:43.024338 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305930d3-73bf-48d8-9f6f-584ffe5b234a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:43.061260 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-305930d3-73bf-48d8-9f6f-584ffe5b234a None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:43.064292 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-305930d3-73bf-48d8-9f6f-584ffe5b234a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:43.080122 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:43.080346 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1094/2149] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:43 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:43.081391 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305930d3-73bf-48d8-9f6f-584ffe5b234a None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:43.092384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305930d3-73bf-48d8-9f6f-584ffe5b234a None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:43.101777 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76adaa29-8b17-4641-951f-eb673d3e2eeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:43.102896 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76adaa29-8b17-4641-951f-eb673d3e2eeb admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:43.103491 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76adaa29-8b17-4641-951f-eb673d3e2eeb admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:43.104076 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76adaa29-8b17-4641-951f-eb673d3e2eeb admin admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:43.105862 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76adaa29-8b17-4641-951f-eb673d3e2eeb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:41:43.106396 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76adaa29-8b17-4641-951f-eb673d3e2eeb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:43.109350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76adaa29-8b17-4641-951f-eb673d3e2eeb admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:43.127818 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1095/2150] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:41:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:41:43.129190 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76adaa29-8b17-4641-951f-eb673d3e2eeb admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:43.145735 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76adaa29-8b17-4641-951f-eb673d3e2eeb admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:43.153543 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc944b2-484e-4e4c-8524-cf2835790563 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:43.154557 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc944b2-484e-4e4c-8524-cf2835790563 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:43.155067 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc944b2-484e-4e4c-8524-cf2835790563 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:43.155558 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc944b2-484e-4e4c-8524-cf2835790563 admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:43.157509 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc944b2-484e-4e4c-8524-cf2835790563 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:43.159701 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc944b2-484e-4e4c-8524-cf2835790563 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:43.182103 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-686b66d3-82df-44d9-8711-c487f910842c None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:43.196175 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2536030c-902a-4bb9-927f-f2acb682d412 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:43.197652 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2536030c-902a-4bb9-927f-f2acb682d412 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:43.202599 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2536030c-902a-4bb9-927f-f2acb682d412 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:43.202599 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2536030c-902a-4bb9-927f-f2acb682d412 admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:43.202599 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2536030c-902a-4bb9-927f-f2acb682d412 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:43.204388 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2536030c-902a-4bb9-927f-f2acb682d412 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:43.293744 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1096/2151] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:41:43 2022] POST /identity/v3/projects => generated 385 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:41:43.293744 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc944b2-484e-4e4c-8524-cf2835790563 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:43.329643 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1056/2152] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:41:42 2022] POST /identity/v3/projects => generated 381 bytes in 446 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:41:43.330917 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2536030c-902a-4bb9-927f-f2acb682d412 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:43.456783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc944b2-484e-4e4c-8524-cf2835790563 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:43.462011 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2536030c-902a-4bb9-927f-f2acb682d412 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:43.470126 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e770e1-c27f-47f2-bef9-aee3456e2b21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:43.471751 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e770e1-c27f-47f2-bef9-aee3456e2b21 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:43.472022 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e770e1-c27f-47f2-bef9-aee3456e2b21 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:43.472307 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e770e1-c27f-47f2-bef9-aee3456e2b21 admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:43.475335 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e770e1-c27f-47f2-bef9-aee3456e2b21 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:43.477685 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e770e1-c27f-47f2-bef9-aee3456e2b21 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:43.479711 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f2eae6-ca01-4967-b88f-11637c1a47c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:43.480411 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f2eae6-ca01-4967-b88f-11637c1a47c5 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:43.480704 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f2eae6-ca01-4967-b88f-11637c1a47c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:43.481136 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f2eae6-ca01-4967-b88f-11637c1a47c5 admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:43.481935 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f2eae6-ca01-4967-b88f-11637c1a47c5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:43.484033 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f2eae6-ca01-4967-b88f-11637c1a47c5 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:43.591201 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1097/2153] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:41:43 2022] POST /identity/v3/users => generated 293 bytes in 299 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:41:43.592324 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e770e1-c27f-47f2-bef9-aee3456e2b21 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:43.596281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1057/2154] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:41:43 2022] POST /identity/v3/users => generated 297 bytes in 266 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:41:43.599949 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f2eae6-ca01-4967-b88f-11637c1a47c5 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:43.605194 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e770e1-c27f-47f2-bef9-aee3456e2b21 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:43.611443 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf57d495-fa13-4e3c-a2b9-4071eae7e840 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:43.612343 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf57d495-fa13-4e3c-a2b9-4071eae7e840 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:43.612658 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf57d495-fa13-4e3c-a2b9-4071eae7e840 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:43.613061 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf57d495-fa13-4e3c-a2b9-4071eae7e840 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:43.614042 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf57d495-fa13-4e3c-a2b9-4071eae7e840 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:43.614195 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f2eae6-ca01-4967-b88f-11637c1a47c5 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:43.615983 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf57d495-fa13-4e3c-a2b9-4071eae7e840 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:43.620769 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab89e2c0-32f8-4c37-bdf2-4a5aefdb0cf4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:43.621715 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab89e2c0-32f8-4c37-bdf2-4a5aefdb0cf4 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:43.626349 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab89e2c0-32f8-4c37-bdf2-4a5aefdb0cf4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:43.627162 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab89e2c0-32f8-4c37-bdf2-4a5aefdb0cf4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:43.629245 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab89e2c0-32f8-4c37-bdf2-4a5aefdb0cf4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:43.631455 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab89e2c0-32f8-4c37-bdf2-4a5aefdb0cf4 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:43.646064 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1058/2155] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:43 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 12 17:41:43.646976 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1098/2156] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:43 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:41:43.649561 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab89e2c0-32f8-4c37-bdf2-4a5aefdb0cf4 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:43.650630 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf57d495-fa13-4e3c-a2b9-4071eae7e840 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:43.664529 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf57d495-fa13-4e3c-a2b9-4071eae7e840 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:43.665687 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab89e2c0-32f8-4c37-bdf2-4a5aefdb0cf4 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:43.670755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4696dd71-3472-44cf-955e-7a2dea3d790d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:43.671767 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4696dd71-3472-44cf-955e-7a2dea3d790d admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:43.672241 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4696dd71-3472-44cf-955e-7a2dea3d790d admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:43.672687 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4696dd71-3472-44cf-955e-7a2dea3d790d admin admin] PATH_INFO: `/v3/projects/d8bfbd9e91254001a2c259c3ffd8a7c6/users/ceb901d9ac124947a065452e99d75b68/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:43.676010 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa2264e-7eeb-46c8-bec8-8ca0bfeee72c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:43.677319 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa2264e-7eeb-46c8-bec8-8ca0bfeee72c admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:43.678003 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa2264e-7eeb-46c8-bec8-8ca0bfeee72c admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:43.678538 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa2264e-7eeb-46c8-bec8-8ca0bfeee72c admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:43.679733 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa2264e-7eeb-46c8-bec8-8ca0bfeee72c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:43.682540 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa2264e-7eeb-46c8-bec8-8ca0bfeee72c admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:43.692528 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4696dd71-3472-44cf-955e-7a2dea3d790d admin admin] RBAC: Authorizing `identity:create_grant(project_id=d8bfbd9e91254001a2c259c3ffd8a7c6, user_id=ceb901d9ac124947a065452e99d75b68, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:43.696118 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4696dd71-3472-44cf-955e-7a2dea3d790d admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:43.700325 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1059/2157] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:43 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 12 17:41:43.707935 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa2264e-7eeb-46c8-bec8-8ca0bfeee72c admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:43.723695 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa2264e-7eeb-46c8-bec8-8ca0bfeee72c admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:43.729682 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b81c6c-4a06-44ff-8cb2-5e6acccba438 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:43.730742 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b81c6c-4a06-44ff-8cb2-5e6acccba438 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:43.731113 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b81c6c-4a06-44ff-8cb2-5e6acccba438 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:43.731530 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b81c6c-4a06-44ff-8cb2-5e6acccba438 admin admin] PATH_INFO: `/v3/projects/2125ad9000c94ddd8882f0350d243c8c/users/1fa4f6a9e3644833adbbf7bbbb7c21ae/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:43.752718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b81c6c-4a06-44ff-8cb2-5e6acccba438 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2125ad9000c94ddd8882f0350d243c8c, user_id=1fa4f6a9e3644833adbbf7bbbb7c21ae, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:43.755467 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b81c6c-4a06-44ff-8cb2-5e6acccba438 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:43.792000 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1099/2158] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:41:43 2022] PUT /identity/v3/projects/d8bfbd9e91254001a2c259c3ffd8a7c6/users/ceb901d9ac124947a065452e99d75b68/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:43.795796 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4696dd71-3472-44cf-955e-7a2dea3d790d admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:43.812490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:43.858479 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1060/2159] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:41:43 2022] PUT /identity/v3/projects/2125ad9000c94ddd8882f0350d243c8c/users/1fa4f6a9e3644833adbbf7bbbb7c21ae/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:43.871524 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e990d0a-1934-4cbe-940a-aff51d58b7bb None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:43.872269 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e990d0a-1934-4cbe-940a-aff51d58b7bb None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:43.872892 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e990d0a-1934-4cbe-940a-aff51d58b7bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:43.900305 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4696dd71-3472-44cf-955e-7a2dea3d790d admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:43.916117 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-4e990d0a-1934-4cbe-940a-aff51d58b7bb None None] MFA Rules not processed for user `1fa4f6a9e3644833adbbf7bbbb7c21ae`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:43.917386 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-4e990d0a-1934-4cbe-940a-aff51d58b7bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:44.037518 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4568af3-7baf-4945-aa9f-2c22b0cd5548 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:44.038172 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4568af3-7baf-4945-aa9f-2c22b0cd5548 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:44.038708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4568af3-7baf-4945-aa9f-2c22b0cd5548 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:44.038887 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4568af3-7baf-4945-aa9f-2c22b0cd5548 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:44.039777 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4568af3-7baf-4945-aa9f-2c22b0cd5548 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:44.041069 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1061/2160] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:43 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:44.042188 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4568af3-7baf-4945-aa9f-2c22b0cd5548 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:44.056733 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e990d0a-1934-4cbe-940a-aff51d58b7bb None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:44.058625 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-4e990d0a-1934-4cbe-940a-aff51d58b7bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:44.073621 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:44.073871 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1100/2161] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:43 2022] GET /identity/v3/roles => generated 1420 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 12 17:41:44.079787 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4568af3-7baf-4945-aa9f-2c22b0cd5548 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:44.096798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4568af3-7baf-4945-aa9f-2c22b0cd5548 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:44.107302 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f60949ff-27d6-43c3-bacd-e8313d61939f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:44.108666 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60949ff-27d6-43c3-bacd-e8313d61939f admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:44.108946 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60949ff-27d6-43c3-bacd-e8313d61939f admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:44.109333 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f60949ff-27d6-43c3-bacd-e8313d61939f admin admin] PATH_INFO: `/v3/projects/d8bfbd9e91254001a2c259c3ffd8a7c6/users/ceb901d9ac124947a065452e99d75b68/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:44.114169 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f60949ff-27d6-43c3-bacd-e8313d61939f admin admin] RBAC: Authorizing `identity:create_grant(project_id=d8bfbd9e91254001a2c259c3ffd8a7c6, user_id=ceb901d9ac124947a065452e99d75b68, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:44.118560 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f60949ff-27d6-43c3-bacd-e8313d61939f admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:44.142764 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1101/2162] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:41:44 2022] PUT /identity/v3/projects/d8bfbd9e91254001a2c259c3ffd8a7c6/users/ceb901d9ac124947a065452e99d75b68/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:41:44.153589 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316e655d-3183-4916-bcbf-dfb9d18c3786 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:44.153823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316e655d-3183-4916-bcbf-dfb9d18c3786 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:44.154266 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-316e655d-3183-4916-bcbf-dfb9d18c3786 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:44.191885 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e990d0a-1934-4cbe-940a-aff51d58b7bb None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:44.203464 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-316e655d-3183-4916-bcbf-dfb9d18c3786 None None] MFA Rules not processed for user `ceb901d9ac124947a065452e99d75b68`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:44.204678 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-316e655d-3183-4916-bcbf-dfb9d18c3786 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:44.290770 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba27e5d-e5bf-4a53-bed5-2d22388dee7e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:44.291832 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba27e5d-e5bf-4a53-bed5-2d22388dee7e service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:44.292128 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba27e5d-e5bf-4a53-bed5-2d22388dee7e service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:44.292412 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba27e5d-e5bf-4a53-bed5-2d22388dee7e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:44.324837 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1102/2163] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:44 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:44.339014 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316e655d-3183-4916-bcbf-dfb9d18c3786 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:44.360350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-316e655d-3183-4916-bcbf-dfb9d18c3786 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:44.378643 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-316e655d-3183-4916-bcbf-dfb9d18c3786 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:44.384070 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2fbad38-f454-4041-858c-d8ee18291bac service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:44.385046 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2fbad38-f454-4041-858c-d8ee18291bac service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:44.385584 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2fbad38-f454-4041-858c-d8ee18291bac service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:44.385724 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2fbad38-f454-4041-858c-d8ee18291bac service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:44.397927 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba27e5d-e5bf-4a53-bed5-2d22388dee7e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:44.399441 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2fbad38-f454-4041-858c-d8ee18291bac service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:44.399549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba27e5d-e5bf-4a53-bed5-2d22388dee7e service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:44.403417 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2fbad38-f454-4041-858c-d8ee18291bac service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:44.415971 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1062/2164] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:41:44 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:41:44.428294 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1103/2165] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:41:44 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:41:44.446009 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba27e5d-e5bf-4a53-bed5-2d22388dee7e service cinder] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:44.588102 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba27e5d-e5bf-4a53-bed5-2d22388dee7e service cinder] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:44.593562 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e69f761-795b-4a65-9b57-96d1375a7ee7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:44.594738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e69f761-795b-4a65-9b57-96d1375a7ee7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:44.595060 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e69f761-795b-4a65-9b57-96d1375a7ee7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:44.595349 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e69f761-795b-4a65-9b57-96d1375a7ee7 admin admin] PATH_INFO: `/v3/users/1fa4f6a9e3644833adbbf7bbbb7c21ae` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:44.597134 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e69f761-795b-4a65-9b57-96d1375a7ee7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1fa4f6a9e3644833adbbf7bbbb7c21ae)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:44.599400 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e69f761-795b-4a65-9b57-96d1375a7ee7 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:44.666843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:44.962823 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-6e69f761-795b-4a65-9b57-96d1375a7ee7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1fa4f6a9e3644833adbbf7bbbb7c21ae'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:45.035006 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-6e69f761-795b-4a65-9b57-96d1375a7ee7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1fa4f6a9e3644833adbbf7bbbb7c21ae'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:45.035985 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-6e69f761-795b-4a65-9b57-96d1375a7ee7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1fa4f6a9e3644833adbbf7bbbb7c21ae'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:45.046342 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-6e69f761-795b-4a65-9b57-96d1375a7ee7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1fa4f6a9e3644833adbbf7bbbb7c21ae'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:45.071257 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-6e69f761-795b-4a65-9b57-96d1375a7ee7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1fa4f6a9e3644833adbbf7bbbb7c21ae'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:45.082167 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1063/2166] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:41:44 2022] DELETE /identity/v3/users/1fa4f6a9e3644833adbbf7bbbb7c21ae => generated 0 bytes in 637 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:45.087751 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c075aa-b91a-43a7-9ccc-06758060756f None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:45.088958 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c075aa-b91a-43a7-9ccc-06758060756f None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:45.089936 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c075aa-b91a-43a7-9ccc-06758060756f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:45.131564 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-50c075aa-b91a-43a7-9ccc-06758060756f None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:45.133749 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-50c075aa-b91a-43a7-9ccc-06758060756f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:45.150655 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:45.185045 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e69f761-795b-4a65-9b57-96d1375a7ee7 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:45.186914 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-6e69f761-795b-4a65-9b57-96d1375a7ee7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:45.250533 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1104/2167] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:45.267706 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c075aa-b91a-43a7-9ccc-06758060756f None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:45.270266 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-50c075aa-b91a-43a7-9ccc-06758060756f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:45.322747 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e69f761-795b-4a65-9b57-96d1375a7ee7 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:45.340970 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665e50da-198b-432e-8e58-19d1f65026c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:45.342486 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665e50da-198b-432e-8e58-19d1f65026c8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:45.342981 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665e50da-198b-432e-8e58-19d1f65026c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:45.343451 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665e50da-198b-432e-8e58-19d1f65026c8 admin admin] PATH_INFO: `/v3/users/cfd0ef55a4fe45559b3b8834ad8d5557` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:45.346080 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665e50da-198b-432e-8e58-19d1f65026c8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cfd0ef55a4fe45559b3b8834ad8d5557)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:45.348444 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665e50da-198b-432e-8e58-19d1f65026c8 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:45.386408 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c075aa-b91a-43a7-9ccc-06758060756f None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:45.392692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a52c6611-8b84-43d4-9b12-1721b4fe1ee4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:45.393445 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52c6611-8b84-43d4-9b12-1721b4fe1ee4 service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:45.393842 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52c6611-8b84-43d4-9b12-1721b4fe1ee4 service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:45.394145 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a52c6611-8b84-43d4-9b12-1721b4fe1ee4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:45.417447 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a52c6611-8b84-43d4-9b12-1721b4fe1ee4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:45.421142 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a52c6611-8b84-43d4-9b12-1721b4fe1ee4 service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:45.443627 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1105/2168] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:41:45 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 176 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:41:45.577268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-665e50da-198b-432e-8e58-19d1f65026c8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cfd0ef55a4fe45559b3b8834ad8d5557'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:45.679140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-665e50da-198b-432e-8e58-19d1f65026c8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cfd0ef55a4fe45559b3b8834ad8d5557'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:45.680270 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-665e50da-198b-432e-8e58-19d1f65026c8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cfd0ef55a4fe45559b3b8834ad8d5557'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:45.703545 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-665e50da-198b-432e-8e58-19d1f65026c8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cfd0ef55a4fe45559b3b8834ad8d5557'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:45.724825 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-665e50da-198b-432e-8e58-19d1f65026c8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cfd0ef55a4fe45559b3b8834ad8d5557'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:45.736843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1064/2169] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:41:45 2022] DELETE /identity/v3/users/cfd0ef55a4fe45559b3b8834ad8d5557 => generated 0 bytes in 552 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:45.745276 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cedfff5c-f031-47ae-89ec-60ec032c132e None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:45.746693 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cedfff5c-f031-47ae-89ec-60ec032c132e None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:45.747268 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cedfff5c-f031-47ae-89ec-60ec032c132e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:45.783371 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-cedfff5c-f031-47ae-89ec-60ec032c132e None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:45.785178 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-cedfff5c-f031-47ae-89ec-60ec032c132e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:45.930194 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1106/2170] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:45.947076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665e50da-198b-432e-8e58-19d1f65026c8 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:45.948612 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-665e50da-198b-432e-8e58-19d1f65026c8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:45.974627 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:DELETE=19,keystone:SELECT=79,keystone:INSERT=2 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:46.047132 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665e50da-198b-432e-8e58-19d1f65026c8 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:46.071457 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdcfb31c-a6eb-46c1-8b2e-afa1df270d45 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:46.072150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcfb31c-a6eb-46c1-8b2e-afa1df270d45 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:46.073784 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcfb31c-a6eb-46c1-8b2e-afa1df270d45 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:46.073784 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdcfb31c-a6eb-46c1-8b2e-afa1df270d45 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:46.108261 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdcfb31c-a6eb-46c1-8b2e-afa1df270d45 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:46.110461 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdcfb31c-a6eb-46c1-8b2e-afa1df270d45 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:46.127651 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1065/2171] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:41:45 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:41:46.293198 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cedfff5c-f031-47ae-89ec-60ec032c132e None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:46.294890 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-cedfff5c-f031-47ae-89ec-60ec032c132e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:46.311216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cedfff5c-f031-47ae-89ec-60ec032c132e None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:46.312355 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6cda74-49f7-4500-af84-25f42fbf0793 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:46.313785 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6cda74-49f7-4500-af84-25f42fbf0793 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:46.314074 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6cda74-49f7-4500-af84-25f42fbf0793 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:46.314355 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6cda74-49f7-4500-af84-25f42fbf0793 admin admin] PATH_INFO: `/v3/projects/2125ad9000c94ddd8882f0350d243c8c` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:46.318448 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6cda74-49f7-4500-af84-25f42fbf0793 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2125ad9000c94ddd8882f0350d243c8c)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:46.320435 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6cda74-49f7-4500-af84-25f42fbf0793 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:46.464203 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ad6cda74-49f7-4500-af84-25f42fbf0793 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2125ad9000c94ddd8882f0350d243c8c', 'request_id': 'req-ad6cda74-49f7-4500-af84-25f42fbf0793'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:46.471456 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:46.480662 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ad6cda74-49f7-4500-af84-25f42fbf0793 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2125ad9000c94ddd8882f0350d243c8c', 'request_id': 'req-ad6cda74-49f7-4500-af84-25f42fbf0793'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:46.583058 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ad6cda74-49f7-4500-af84-25f42fbf0793 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2125ad9000c94ddd8882f0350d243c8c', 'request_id': 'req-ad6cda74-49f7-4500-af84-25f42fbf0793'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:46.604750 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ad6cda74-49f7-4500-af84-25f42fbf0793 admin admin] The token cache is being invalidate because project 2125ad9000c94ddd8882f0350d243c8c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:41:46.605060 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ad6cda74-49f7-4500-af84-25f42fbf0793 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:46.605792 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ad6cda74-49f7-4500-af84-25f42fbf0793 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:46.608150 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1107/2172] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:41:46 2022] DELETE /identity/v3/projects/2125ad9000c94ddd8882f0350d243c8c => generated 0 bytes in 315 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:46.615620 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7d92f5-5bfe-42b5-96a6-85d993112047 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:46.616279 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7d92f5-5bfe-42b5-96a6-85d993112047 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:46.616279 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7d92f5-5bfe-42b5-96a6-85d993112047 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:46.655310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-ef7d92f5-5bfe-42b5-96a6-85d993112047 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:46.657736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-ef7d92f5-5bfe-42b5-96a6-85d993112047 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:46.755014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1066/2173] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:46.768851 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0406aac-5880-4a27-abd8-0bf14a9f917c None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:46.769195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0406aac-5880-4a27-abd8-0bf14a9f917c None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:46.769485 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0406aac-5880-4a27-abd8-0bf14a9f917c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:46.794296 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-a0406aac-5880-4a27-abd8-0bf14a9f917c None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:46.794296 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-a0406aac-5880-4a27-abd8-0bf14a9f917c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:46.802254 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1108/2174] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:41:46.807017 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7d92f5-5bfe-42b5-96a6-85d993112047 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:46.820031 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7d92f5-5bfe-42b5-96a6-85d993112047 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:46.828882 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281bb69c-4c84-4997-87c4-5f74d2a16d8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:46.828882 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281bb69c-4c84-4997-87c4-5f74d2a16d8d admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:46.828882 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281bb69c-4c84-4997-87c4-5f74d2a16d8d admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:46.828882 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281bb69c-4c84-4997-87c4-5f74d2a16d8d admin admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:46.829613 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-281bb69c-4c84-4997-87c4-5f74d2a16d8d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:41:46.829613 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-281bb69c-4c84-4997-87c4-5f74d2a16d8d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:46.830788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-281bb69c-4c84-4997-87c4-5f74d2a16d8d admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:46.838612 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0406aac-5880-4a27-abd8-0bf14a9f917c None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:46.838612 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-a0406aac-5880-4a27-abd8-0bf14a9f917c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:46.861292 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1067/2175] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:41:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:41:46.862824 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281bb69c-4c84-4997-87c4-5f74d2a16d8d admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:46.868198 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0406aac-5880-4a27-abd8-0bf14a9f917c None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:46.882703 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-281bb69c-4c84-4997-87c4-5f74d2a16d8d admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:46.890295 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e039220-627e-42b6-a401-4c95915cbba0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:46.891766 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e039220-627e-42b6-a401-4c95915cbba0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:46.892282 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e039220-627e-42b6-a401-4c95915cbba0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:46.893443 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e039220-627e-42b6-a401-4c95915cbba0 admin admin] PATH_INFO: `/v3/projects/11a327f750534f3b8575a5f17bde34be` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:46.895540 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e039220-627e-42b6-a401-4c95915cbba0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=11a327f750534f3b8575a5f17bde34be)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:46.897641 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e039220-627e-42b6-a401-4c95915cbba0 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:46.906371 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4c35aa-871f-4be8-9c01-c3e97b021e95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:46.911493 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4c35aa-871f-4be8-9c01-c3e97b021e95 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:46.911493 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4c35aa-871f-4be8-9c01-c3e97b021e95 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:46.912449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4c35aa-871f-4be8-9c01-c3e97b021e95 admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:46.913817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d4c35aa-871f-4be8-9c01-c3e97b021e95 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:46.917698 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d4c35aa-871f-4be8-9c01-c3e97b021e95 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:47.075527 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-9e039220-627e-42b6-a401-4c95915cbba0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '11a327f750534f3b8575a5f17bde34be', 'request_id': 'req-9e039220-627e-42b6-a401-4c95915cbba0'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:47.091460 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-9e039220-627e-42b6-a401-4c95915cbba0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '11a327f750534f3b8575a5f17bde34be', 'request_id': 'req-9e039220-627e-42b6-a401-4c95915cbba0'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:47.109582 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1068/2176] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:41:46 2022] POST /identity/v3/projects => generated 373 bytes in 245 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:41:47.112470 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4c35aa-871f-4be8-9c01-c3e97b021e95 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:47.176163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-9e039220-627e-42b6-a401-4c95915cbba0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '11a327f750534f3b8575a5f17bde34be', 'request_id': 'req-9e039220-627e-42b6-a401-4c95915cbba0'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:47.214227 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-9e039220-627e-42b6-a401-4c95915cbba0 admin admin] The token cache is being invalidate because project 11a327f750534f3b8575a5f17bde34be was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:41:47.215088 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-9e039220-627e-42b6-a401-4c95915cbba0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:47.216892 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-9e039220-627e-42b6-a401-4c95915cbba0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:47.222188 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1109/2177] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:41:46 2022] DELETE /identity/v3/projects/11a327f750534f3b8575a5f17bde34be => generated 0 bytes in 391 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:47.226860 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e039220-627e-42b6-a401-4c95915cbba0 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:47.230747 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-9e039220-627e-42b6-a401-4c95915cbba0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:47.321987 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4c35aa-871f-4be8-9c01-c3e97b021e95 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:47.338308 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f069807b-cc45-4231-872c-bc445f18e304 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:47.339257 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f069807b-cc45-4231-872c-bc445f18e304 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:47.339707 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f069807b-cc45-4231-872c-bc445f18e304 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:47.343200 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f069807b-cc45-4231-872c-bc445f18e304 admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:47.347892 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f069807b-cc45-4231-872c-bc445f18e304 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:47.364735 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f069807b-cc45-4231-872c-bc445f18e304 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:47.458591 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1069/2178] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:41:47 2022] POST /identity/v3/users => generated 287 bytes in 347 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:41:47.465045 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e039220-627e-42b6-a401-4c95915cbba0 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:47.473954 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b11e60-dd4d-42b3-ba02-4cb1262964b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:47.474355 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f069807b-cc45-4231-872c-bc445f18e304 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:47.475264 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b11e60-dd4d-42b3-ba02-4cb1262964b4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:47.475720 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b11e60-dd4d-42b3-ba02-4cb1262964b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:47.476002 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b11e60-dd4d-42b3-ba02-4cb1262964b4 admin admin] PATH_INFO: `/v3/users/ceb901d9ac124947a065452e99d75b68` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:47.477376 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b11e60-dd4d-42b3-ba02-4cb1262964b4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ceb901d9ac124947a065452e99d75b68)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:47.479878 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b11e60-dd4d-42b3-ba02-4cb1262964b4 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:47.480148 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-f069807b-cc45-4231-872c-bc445f18e304 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:47.544616 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f069807b-cc45-4231-872c-bc445f18e304 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:47.560443 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6df339-86d4-498c-bb85-d4355abb4070 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:47.561643 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6df339-86d4-498c-bb85-d4355abb4070 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:47.562088 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6df339-86d4-498c-bb85-d4355abb4070 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:47.562972 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6df339-86d4-498c-bb85-d4355abb4070 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:47.564512 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6df339-86d4-498c-bb85-d4355abb4070 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:47.567775 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6df339-86d4-498c-bb85-d4355abb4070 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:47.578813 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:47.588893 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1070/2179] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:47 2022] GET /identity/v3/roles => generated 1420 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:41:47.593384 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6df339-86d4-498c-bb85-d4355abb4070 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:47.605257 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6df339-86d4-498c-bb85-d4355abb4070 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:47.617231 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febef9e6-e486-4adc-ba0b-9779f5467559 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:47.622125 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febef9e6-e486-4adc-ba0b-9779f5467559 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:47.622125 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febef9e6-e486-4adc-ba0b-9779f5467559 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:47.622125 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febef9e6-e486-4adc-ba0b-9779f5467559 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:47.622125 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-febef9e6-e486-4adc-ba0b-9779f5467559 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:47.623723 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-febef9e6-e486-4adc-ba0b-9779f5467559 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:47.646468 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1071/2180] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:47 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:41:47.647487 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febef9e6-e486-4adc-ba0b-9779f5467559 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:47.662734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febef9e6-e486-4adc-ba0b-9779f5467559 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:47.670045 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6afb680f-2cbb-41d9-90ea-3b3f802f6b96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:47.671123 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afb680f-2cbb-41d9-90ea-3b3f802f6b96 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:47.671469 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afb680f-2cbb-41d9-90ea-3b3f802f6b96 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:47.671830 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6afb680f-2cbb-41d9-90ea-3b3f802f6b96 admin admin] PATH_INFO: `/v3/projects/4d8a513be02b4337883b0b91e5c85919/users/a7b2735623734176b1d876b4e249f858/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:47.689789 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6afb680f-2cbb-41d9-90ea-3b3f802f6b96 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4d8a513be02b4337883b0b91e5c85919, user_id=a7b2735623734176b1d876b4e249f858, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:47.693005 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6afb680f-2cbb-41d9-90ea-3b3f802f6b96 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:47.805442 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1072/2181] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:41:47 2022] PUT /identity/v3/projects/4d8a513be02b4337883b0b91e5c85919/users/a7b2735623734176b1d876b4e249f858/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:47.818653 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40db42ba-ed17-4f6c-9400-275ba18d0d97 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:47.819542 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40db42ba-ed17-4f6c-9400-275ba18d0d97 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:47.820091 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40db42ba-ed17-4f6c-9400-275ba18d0d97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:47.830382 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-c3b11e60-dd4d-42b3-ba02-4cb1262964b4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ceb901d9ac124947a065452e99d75b68'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:47.831319 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-c3b11e60-dd4d-42b3-ba02-4cb1262964b4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ceb901d9ac124947a065452e99d75b68'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:47.832104 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-c3b11e60-dd4d-42b3-ba02-4cb1262964b4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ceb901d9ac124947a065452e99d75b68'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:47.843594 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-c3b11e60-dd4d-42b3-ba02-4cb1262964b4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ceb901d9ac124947a065452e99d75b68'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:47.847209 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:47.863787 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-c3b11e60-dd4d-42b3-ba02-4cb1262964b4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ceb901d9ac124947a065452e99d75b68'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:47.885683 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-40db42ba-ed17-4f6c-9400-275ba18d0d97 None None] MFA Rules not processed for user `a7b2735623734176b1d876b4e249f858`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:47.890686 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-40db42ba-ed17-4f6c-9400-275ba18d0d97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:47.956069 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1110/2182] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:41:47 2022] DELETE /identity/v3/users/ceb901d9ac124947a065452e99d75b68 => generated 0 bytes in 731 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:48.018831 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1073/2183] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:47 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:48.033340 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b11e60-dd4d-42b3-ba02-4cb1262964b4 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:48.035021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-c3b11e60-dd4d-42b3-ba02-4cb1262964b4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:48.136420 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b11e60-dd4d-42b3-ba02-4cb1262964b4 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:48.141682 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3628029f-3a07-4c13-bac0-4b96c6f599d5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:48.142628 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3628029f-3a07-4c13-bac0-4b96c6f599d5 service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:48.142966 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3628029f-3a07-4c13-bac0-4b96c6f599d5 service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:48.143322 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3628029f-3a07-4c13-bac0-4b96c6f599d5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:48.231219 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3628029f-3a07-4c13-bac0-4b96c6f599d5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:48.236320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3628029f-3a07-4c13-bac0-4b96c6f599d5 service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:48.253232 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1111/2184] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:41:48 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 220 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:41:48.285730 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40db42ba-ed17-4f6c-9400-275ba18d0d97 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:48.287719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-40db42ba-ed17-4f6c-9400-275ba18d0d97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:48.437183 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40db42ba-ed17-4f6c-9400-275ba18d0d97 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:48.603197 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6271f61-960c-4c68-aeb5-b81b9f62ee12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:48.604255 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6271f61-960c-4c68-aeb5-b81b9f62ee12 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:48.604625 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6271f61-960c-4c68-aeb5-b81b9f62ee12 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:48.605012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6271f61-960c-4c68-aeb5-b81b9f62ee12 admin admin] PATH_INFO: `/v3/users/a7b2735623734176b1d876b4e249f858` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:48.606664 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6271f61-960c-4c68-aeb5-b81b9f62ee12 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a7b2735623734176b1d876b4e249f858)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:48.609790 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6271f61-960c-4c68-aeb5-b81b9f62ee12 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:48.613479 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:48.831571 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3628029f-3a07-4c13-bac0-4b96c6f599d5 service cinder] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:48.834903 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-3628029f-3a07-4c13-bac0-4b96c6f599d5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:48.881126 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3628029f-3a07-4c13-bac0-4b96c6f599d5 service cinder] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:48.887587 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11004b2a-f9cc-42cf-b90b-bcc19a25a711 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:48.898116 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11004b2a-f9cc-42cf-b90b-bcc19a25a711 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:48.898116 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11004b2a-f9cc-42cf-b90b-bcc19a25a711 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:48.898116 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11004b2a-f9cc-42cf-b90b-bcc19a25a711 admin admin] PATH_INFO: `/v3/projects/d8bfbd9e91254001a2c259c3ffd8a7c6` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:48.898116 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11004b2a-f9cc-42cf-b90b-bcc19a25a711 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d8bfbd9e91254001a2c259c3ffd8a7c6)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:48.901272 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11004b2a-f9cc-42cf-b90b-bcc19a25a711 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:48.918060 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-a6271f61-960c-4c68-aeb5-b81b9f62ee12 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a7b2735623734176b1d876b4e249f858'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:49.069425 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-a6271f61-960c-4c68-aeb5-b81b9f62ee12 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a7b2735623734176b1d876b4e249f858'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:49.070793 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-a6271f61-960c-4c68-aeb5-b81b9f62ee12 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a7b2735623734176b1d876b4e249f858'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:49.100833 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-a6271f61-960c-4c68-aeb5-b81b9f62ee12 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a7b2735623734176b1d876b4e249f858'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:49.125896 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-a6271f61-960c-4c68-aeb5-b81b9f62ee12 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a7b2735623734176b1d876b4e249f858'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:49.126989 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-11004b2a-f9cc-42cf-b90b-bcc19a25a711 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd8bfbd9e91254001a2c259c3ffd8a7c6', 'request_id': 'req-11004b2a-f9cc-42cf-b90b-bcc19a25a711'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:49.136298 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1074/2185] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:41:48 2022] DELETE /identity/v3/users/a7b2735623734176b1d876b4e249f858 => generated 0 bytes in 851 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:41:49.142699 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2593ec92-4032-45be-b041-e6703bf3c1b2 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:49.143492 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2593ec92-4032-45be-b041-e6703bf3c1b2 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:49.144020 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2593ec92-4032-45be-b041-e6703bf3c1b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:49.144865 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-11004b2a-f9cc-42cf-b90b-bcc19a25a711 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd8bfbd9e91254001a2c259c3ffd8a7c6', 'request_id': 'req-11004b2a-f9cc-42cf-b90b-bcc19a25a711'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:49.181128 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-2593ec92-4032-45be-b041-e6703bf3c1b2 None None] MFA Rules not processed for user `3ac0d5d8de4943a3888abee3b24b8a6f`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:49.184597 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-2593ec92-4032-45be-b041-e6703bf3c1b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:49.210258 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-11004b2a-f9cc-42cf-b90b-bcc19a25a711 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd8bfbd9e91254001a2c259c3ffd8a7c6', 'request_id': 'req-11004b2a-f9cc-42cf-b90b-bcc19a25a711'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:49.246823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-11004b2a-f9cc-42cf-b90b-bcc19a25a711 admin admin] The token cache is being invalidate because project d8bfbd9e91254001a2c259c3ffd8a7c6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:41:49.247598 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-11004b2a-f9cc-42cf-b90b-bcc19a25a711 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:49.248769 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-11004b2a-f9cc-42cf-b90b-bcc19a25a711 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:49.262090 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1112/2186] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:41:48 2022] DELETE /identity/v3/projects/d8bfbd9e91254001a2c259c3ffd8a7c6 => generated 0 bytes in 425 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:49.264261 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1df6b4-c041-4d61-bd1c-aa642fd3f7f2 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:49.265910 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1df6b4-c041-4d61-bd1c-aa642fd3f7f2 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:49.265910 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1df6b4-c041-4d61-bd1c-aa642fd3f7f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:49.288284 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-ae1df6b4-c041-4d61-bd1c-aa642fd3f7f2 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:49.290050 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-ae1df6b4-c041-4d61-bd1c-aa642fd3f7f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:49.306657 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:49.309423 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1075/2187] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Jul 12 17:41:49 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:49.311133 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5155aa-71ed-4249-bf7c-4e1a35c701bb None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:49.311606 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5155aa-71ed-4249-bf7c-4e1a35c701bb None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:49.312056 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5155aa-71ed-4249-bf7c-4e1a35c701bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:49.340840 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-9f5155aa-71ed-4249-bf7c-4e1a35c701bb None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:49.342299 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-9f5155aa-71ed-4249-bf7c-4e1a35c701bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:49.435620 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1113/2188] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:49.436202 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1076/2189] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:49.436741 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1df6b4-c041-4d61-bd1c-aa642fd3f7f2 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:49.439175 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-ae1df6b4-c041-4d61-bd1c-aa642fd3f7f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:49.454805 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e69a20-a631-4e58-94c6-f6027507c1e0 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:49.455638 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e69a20-a631-4e58-94c6-f6027507c1e0 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:49.456108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e69a20-a631-4e58-94c6-f6027507c1e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:49.479081 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-b6e69a20-a631-4e58-94c6-f6027507c1e0 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:49.480662 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-b6e69a20-a631-4e58-94c6-f6027507c1e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:49.491434 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1077/2190] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:49.492562 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e69a20-a631-4e58-94c6-f6027507c1e0 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:49.494658 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-b6e69a20-a631-4e58-94c6-f6027507c1e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:49.529704 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:49.658195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1df6b4-c041-4d61-bd1c-aa642fd3f7f2 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:49.662596 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e69a20-a631-4e58-94c6-f6027507c1e0 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:49.666664 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0daca8-b168-4a91-a59e-5f598f9ec04a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7906d1a19c642bd93507a27bf3e3dae', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:49.667580 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0daca8-b168-4a91-a59e-5f598f9ec04a service swift] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:49.667920 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0daca8-b168-4a91-a59e-5f598f9ec04a service swift] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:49.668352 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0daca8-b168-4a91-a59e-5f598f9ec04a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:49.671475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70499774-eb19-4642-8568-9816af395c3b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:49.672886 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70499774-eb19-4642-8568-9816af395c3b service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:49.673167 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70499774-eb19-4642-8568-9816af395c3b service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:49.673494 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70499774-eb19-4642-8568-9816af395c3b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:49.684342 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf0daca8-b168-4a91-a59e-5f598f9ec04a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:49.686594 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70499774-eb19-4642-8568-9816af395c3b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:49.686712 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf0daca8-b168-4a91-a59e-5f598f9ec04a service swift] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:49.686938 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70499774-eb19-4642-8568-9816af395c3b service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:49.702582 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1114/2191] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Jul 12 17:41:49 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 267 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 12 17:41:49.704042 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0daca8-b168-4a91-a59e-5f598f9ec04a service swift] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:49.718878 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1078/2192] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:41:49 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 227 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:41:49.732677 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0daca8-b168-4a91-a59e-5f598f9ec04a service swift] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:49.743019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2a068d-0995-4d9a-ad64-bcd05a17e380 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:49.743790 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2a068d-0995-4d9a-ad64-bcd05a17e380 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:49.743980 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2a068d-0995-4d9a-ad64-bcd05a17e380 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:49.744256 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2a068d-0995-4d9a-ad64-bcd05a17e380 admin admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:49.745019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b2a068d-0995-4d9a-ad64-bcd05a17e380 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:41:49.745395 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b2a068d-0995-4d9a-ad64-bcd05a17e380 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:49.749848 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b2a068d-0995-4d9a-ad64-bcd05a17e380 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:49.771084 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1115/2193] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:41:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:41:49.775771 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70499774-eb19-4642-8568-9816af395c3b service neutron] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:49.790103 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70499774-eb19-4642-8568-9816af395c3b service neutron] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:49.823086 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12cb400c-c0e4-4ab5-8bfc-84109c2649fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:49.825605 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cb400c-c0e4-4ab5-8bfc-84109c2649fe admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:49.825983 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cb400c-c0e4-4ab5-8bfc-84109c2649fe admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:49.826273 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12cb400c-c0e4-4ab5-8bfc-84109c2649fe admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:49.827136 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12cb400c-c0e4-4ab5-8bfc-84109c2649fe admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:49.832750 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12cb400c-c0e4-4ab5-8bfc-84109c2649fe admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:49.972878 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1079/2194] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:41:49 2022] POST /identity/v3/projects => generated 385 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:41:49.977707 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2a068d-0995-4d9a-ad64-bcd05a17e380 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:49.998334 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2a068d-0995-4d9a-ad64-bcd05a17e380 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:50.117726 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c523c217-ece8-4cfb-9dbe-7c9ce7ae2636 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:50.119107 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c523c217-ece8-4cfb-9dbe-7c9ce7ae2636 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:50.119708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c523c217-ece8-4cfb-9dbe-7c9ce7ae2636 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:50.120095 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c523c217-ece8-4cfb-9dbe-7c9ce7ae2636 admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:50.120922 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c523c217-ece8-4cfb-9dbe-7c9ce7ae2636 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:50.123841 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c523c217-ece8-4cfb-9dbe-7c9ce7ae2636 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:50.230768 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1116/2195] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:41:49 2022] POST /identity/v3/users => generated 293 bytes in 254 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:41:50.236338 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12cb400c-c0e4-4ab5-8bfc-84109c2649fe admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:50.257823 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12cb400c-c0e4-4ab5-8bfc-84109c2649fe admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:50.271927 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c5c4e2c-78b7-424c-a313-99a7b1a66718 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:50.272165 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5c4e2c-78b7-424c-a313-99a7b1a66718 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:50.272315 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5c4e2c-78b7-424c-a313-99a7b1a66718 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:50.272601 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5c4e2c-78b7-424c-a313-99a7b1a66718 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:50.273370 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c5c4e2c-78b7-424c-a313-99a7b1a66718 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:50.275040 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c5c4e2c-78b7-424c-a313-99a7b1a66718 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:50.352416 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1080/2196] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:50 2022] GET /identity/v3/roles => generated 1420 bytes in 117 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:41:50.359212 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c523c217-ece8-4cfb-9dbe-7c9ce7ae2636 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:50.374790 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c523c217-ece8-4cfb-9dbe-7c9ce7ae2636 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:50.390394 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b004486d-904a-4b53-9aaf-60819aab9185 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:50.392148 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b004486d-904a-4b53-9aaf-60819aab9185 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:50.393433 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b004486d-904a-4b53-9aaf-60819aab9185 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:50.394058 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b004486d-904a-4b53-9aaf-60819aab9185 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:50.395275 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b004486d-904a-4b53-9aaf-60819aab9185 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:50.397590 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b004486d-904a-4b53-9aaf-60819aab9185 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:50.490731 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c5c4e2c-78b7-424c-a313-99a7b1a66718 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:50.494537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1117/2197] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:50 2022] GET /identity/v3/roles => generated 1420 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:41:50.505080 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c5c4e2c-78b7-424c-a313-99a7b1a66718 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:50.548999 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387c236b-8f5f-4915-80a3-a9ca82f1b634 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:50.548999 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387c236b-8f5f-4915-80a3-a9ca82f1b634 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:50.548999 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387c236b-8f5f-4915-80a3-a9ca82f1b634 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:50.548999 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387c236b-8f5f-4915-80a3-a9ca82f1b634 admin admin] PATH_INFO: `/v3/projects/80285308991b47308701fd3363bec122/users/ed4726967dd844b692fa7badd510267a/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:50.594615 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-387c236b-8f5f-4915-80a3-a9ca82f1b634 admin admin] RBAC: Authorizing `identity:create_grant(project_id=80285308991b47308701fd3363bec122, user_id=ed4726967dd844b692fa7badd510267a, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:50.597741 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-387c236b-8f5f-4915-80a3-a9ca82f1b634 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:50.929352 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b004486d-904a-4b53-9aaf-60819aab9185 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:50.931570 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-b004486d-904a-4b53-9aaf-60819aab9185 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:50.937069 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1081/2198] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:41:50 2022] PUT /identity/v3/projects/80285308991b47308701fd3363bec122/users/ed4726967dd844b692fa7badd510267a/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 444 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:50.939787 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa945ef-ee3f-442e-893b-45f12fcb4e36 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:50.940226 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa945ef-ee3f-442e-893b-45f12fcb4e36 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:50.940820 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa945ef-ee3f-442e-893b-45f12fcb4e36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:50.978161 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b004486d-904a-4b53-9aaf-60819aab9185 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:50.990625 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:51.006012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-bfa945ef-ee3f-442e-893b-45f12fcb4e36 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:51.007876 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-bfa945ef-ee3f-442e-893b-45f12fcb4e36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:51.153852 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b424ac8b-021e-41a7-88dd-d97b44912aa9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:51.158001 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b424ac8b-021e-41a7-88dd-d97b44912aa9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:51.163038 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b424ac8b-021e-41a7-88dd-d97b44912aa9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:51.163398 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b424ac8b-021e-41a7-88dd-d97b44912aa9 admin admin] PATH_INFO: `/v3/projects/4d8a513be02b4337883b0b91e5c85919` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:51.182317 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b424ac8b-021e-41a7-88dd-d97b44912aa9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4d8a513be02b4337883b0b91e5c85919)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:51.191661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b424ac8b-021e-41a7-88dd-d97b44912aa9 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:51.225011 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1082/2199] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 287 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:51.238769 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa945ef-ee3f-442e-893b-45f12fcb4e36 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:51.403641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa945ef-ee3f-442e-893b-45f12fcb4e36 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:51.417673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007aed8f-fb65-4717-a400-c2ec1ab086a6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:51.421613 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007aed8f-fb65-4717-a400-c2ec1ab086a6 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:51.423130 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007aed8f-fb65-4717-a400-c2ec1ab086a6 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:51.423419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007aed8f-fb65-4717-a400-c2ec1ab086a6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:51.490947 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007aed8f-fb65-4717-a400-c2ec1ab086a6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:51.504601 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007aed8f-fb65-4717-a400-c2ec1ab086a6 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:51.520695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-b424ac8b-021e-41a7-88dd-d97b44912aa9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4d8a513be02b4337883b0b91e5c85919', 'request_id': 'req-b424ac8b-021e-41a7-88dd-d97b44912aa9'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:51.547615 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-b424ac8b-021e-41a7-88dd-d97b44912aa9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4d8a513be02b4337883b0b91e5c85919', 'request_id': 'req-b424ac8b-021e-41a7-88dd-d97b44912aa9'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:51.552670 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1083/2200] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:41:51 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:41:51.644441 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-b424ac8b-021e-41a7-88dd-d97b44912aa9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4d8a513be02b4337883b0b91e5c85919', 'request_id': 'req-b424ac8b-021e-41a7-88dd-d97b44912aa9'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:51.667070 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-b424ac8b-021e-41a7-88dd-d97b44912aa9 admin admin] The token cache is being invalidate because project 4d8a513be02b4337883b0b91e5c85919 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:41:51.667390 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-b424ac8b-021e-41a7-88dd-d97b44912aa9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:51.670617 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-b424ac8b-021e-41a7-88dd-d97b44912aa9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:41:51.676239 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1118/2201] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:41:50 2022] DELETE /identity/v3/projects/4d8a513be02b4337883b0b91e5c85919 => generated 0 bytes in 744 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:41:51.681856 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39339ca8-1855-4499-9ff6-bc2a66c54c33 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:51.683575 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39339ca8-1855-4499-9ff6-bc2a66c54c33 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:51.683575 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39339ca8-1855-4499-9ff6-bc2a66c54c33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:51.717887 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-39339ca8-1855-4499-9ff6-bc2a66c54c33 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:51.719778 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-39339ca8-1855-4499-9ff6-bc2a66c54c33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:51.728509 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:51.858411 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1084/2202] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:51.870171 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b41fb8-9e38-480d-adbe-465576491326 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:51.870740 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b41fb8-9e38-480d-adbe-465576491326 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:51.871313 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b41fb8-9e38-480d-adbe-465576491326 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:51.900549 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-e0b41fb8-9e38-480d-adbe-465576491326 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:51.901963 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-e0b41fb8-9e38-480d-adbe-465576491326 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:51.918294 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1119/2203] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:51.924950 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39339ca8-1855-4499-9ff6-bc2a66c54c33 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:51.940619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39339ca8-1855-4499-9ff6-bc2a66c54c33 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:51.973614 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7b5a6d-a8ef-4bc2-890f-4651ae405116 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:51.975279 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7b5a6d-a8ef-4bc2-890f-4651ae405116 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:51.983065 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7b5a6d-a8ef-4bc2-890f-4651ae405116 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:51.983065 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7b5a6d-a8ef-4bc2-890f-4651ae405116 admin admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:51.983065 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df7b5a6d-a8ef-4bc2-890f-4651ae405116 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:41:51.983065 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df7b5a6d-a8ef-4bc2-890f-4651ae405116 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:51.983065 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df7b5a6d-a8ef-4bc2-890f-4651ae405116 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:52.015405 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1085/2204] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:41:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:41:52.024011 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b41fb8-9e38-480d-adbe-465576491326 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:52.044081 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b41fb8-9e38-480d-adbe-465576491326 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:52.064503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb39d42-1cfd-4994-87bd-fad76a3167d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:52.065225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb39d42-1cfd-4994-87bd-fad76a3167d8 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:52.065852 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb39d42-1cfd-4994-87bd-fad76a3167d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:52.066583 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb39d42-1cfd-4994-87bd-fad76a3167d8 admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:52.067335 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb39d42-1cfd-4994-87bd-fad76a3167d8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:52.071561 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb39d42-1cfd-4994-87bd-fad76a3167d8 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:52.195803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1120/2205] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:41:52 2022] POST /identity/v3/projects => generated 377 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:41:52.201061 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7b5a6d-a8ef-4bc2-890f-4651ae405116 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:52.237621 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7b5a6d-a8ef-4bc2-890f-4651ae405116 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:52.422895 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a5f14a-96b8-4306-9eba-e625553230a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:52.423575 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a5f14a-96b8-4306-9eba-e625553230a3 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:52.423874 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a5f14a-96b8-4306-9eba-e625553230a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:52.424167 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a5f14a-96b8-4306-9eba-e625553230a3 admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:52.424907 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a5f14a-96b8-4306-9eba-e625553230a3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:52.427016 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a5f14a-96b8-4306-9eba-e625553230a3 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:52.556558 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1086/2206] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:41:52 2022] POST /identity/v3/users => generated 289 bytes in 356 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:41:52.558585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb39d42-1cfd-4994-87bd-fad76a3167d8 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:52.588231 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb39d42-1cfd-4994-87bd-fad76a3167d8 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:52.606451 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68fd6baf-fd06-4604-b1ef-384e62989f94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:52.607727 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fd6baf-fd06-4604-b1ef-384e62989f94 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:52.608067 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fd6baf-fd06-4604-b1ef-384e62989f94 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:52.608355 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fd6baf-fd06-4604-b1ef-384e62989f94 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:52.609143 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68fd6baf-fd06-4604-b1ef-384e62989f94 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:52.611884 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68fd6baf-fd06-4604-b1ef-384e62989f94 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:52.676921 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1121/2207] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:52 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 12 17:41:52.680293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a5f14a-96b8-4306-9eba-e625553230a3 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:52.699082 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a5f14a-96b8-4306-9eba-e625553230a3 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:52.717792 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2868904f-48b3-4509-b645-6a8aa83218e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:52.719370 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2868904f-48b3-4509-b645-6a8aa83218e3 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:52.721336 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2868904f-48b3-4509-b645-6a8aa83218e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:52.722397 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2868904f-48b3-4509-b645-6a8aa83218e3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:52.734220 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2868904f-48b3-4509-b645-6a8aa83218e3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:52.740526 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2868904f-48b3-4509-b645-6a8aa83218e3 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:52.764241 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1087/2208] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:52 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:41:52.770685 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68fd6baf-fd06-4604-b1ef-384e62989f94 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:52.816421 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68fd6baf-fd06-4604-b1ef-384e62989f94 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:52.841529 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89fa2ea-808f-4079-98a3-2707093e941c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:52.843010 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89fa2ea-808f-4079-98a3-2707093e941c admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:52.844044 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89fa2ea-808f-4079-98a3-2707093e941c admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:52.846549 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89fa2ea-808f-4079-98a3-2707093e941c admin admin] PATH_INFO: `/v3/projects/c0bbd2ab656e48188ef7b9362f01de15/users/74cfb3971bab4c0bb8eff71f13d0e1d8/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:52.908340 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b89fa2ea-808f-4079-98a3-2707093e941c admin admin] RBAC: Authorizing `identity:create_grant(project_id=c0bbd2ab656e48188ef7b9362f01de15, user_id=74cfb3971bab4c0bb8eff71f13d0e1d8, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:52.912478 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b89fa2ea-808f-4079-98a3-2707093e941c admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:53.025276 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1122/2209] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:41:52 2022] PUT /identity/v3/projects/c0bbd2ab656e48188ef7b9362f01de15/users/74cfb3971bab4c0bb8eff71f13d0e1d8/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 257 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:41:53.046396 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a5d621-e848-4216-8c18-d1f06846f8ef None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:53.047447 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a5d621-e848-4216-8c18-d1f06846f8ef None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:53.047648 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a5d621-e848-4216-8c18-d1f06846f8ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:53.153708 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-17a5d621-e848-4216-8c18-d1f06846f8ef None None] MFA Rules not processed for user `74cfb3971bab4c0bb8eff71f13d0e1d8`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:53.164079 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-17a5d621-e848-4216-8c18-d1f06846f8ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:53.390467 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1088/2210] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:53 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 346 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:53.395438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89fa2ea-808f-4079-98a3-2707093e941c admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:53.502621 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:53.562255 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89fa2ea-808f-4079-98a3-2707093e941c admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:53.581576 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1dfd6b-346f-48aa-86bd-4a6bf2da4cf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:53.582831 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1dfd6b-346f-48aa-86bd-4a6bf2da4cf2 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:53.583365 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1dfd6b-346f-48aa-86bd-4a6bf2da4cf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:53.583858 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1dfd6b-346f-48aa-86bd-4a6bf2da4cf2 admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:53.584790 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1dfd6b-346f-48aa-86bd-4a6bf2da4cf2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:53.587210 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1dfd6b-346f-48aa-86bd-4a6bf2da4cf2 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:53.753551 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1123/2211] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:41:53 2022] POST /identity/v3/projects => generated 377 bytes in 359 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:41:53.757826 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a5d621-e848-4216-8c18-d1f06846f8ef None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:53.774605 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:53.785198 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a5d621-e848-4216-8c18-d1f06846f8ef None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:53.934417 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739beb7c-243f-4125-addb-5e91c6f9999d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:53.934417 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739beb7c-243f-4125-addb-5e91c6f9999d admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:53.934417 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739beb7c-243f-4125-addb-5e91c6f9999d admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:53.934417 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739beb7c-243f-4125-addb-5e91c6f9999d admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:53.936444 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739beb7c-243f-4125-addb-5e91c6f9999d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:53.940702 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739beb7c-243f-4125-addb-5e91c6f9999d admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:54.092824 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1089/2212] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:41:53 2022] POST /identity/v3/users => generated 289 bytes in 336 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:41:54.099590 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1dfd6b-346f-48aa-86bd-4a6bf2da4cf2 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:54.141995 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1dfd6b-346f-48aa-86bd-4a6bf2da4cf2 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:54.163353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4d8895-f14f-483a-8f96-ad0dc64e563b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:54.164589 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4d8895-f14f-483a-8f96-ad0dc64e563b admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:54.165109 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4d8895-f14f-483a-8f96-ad0dc64e563b admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:54.165691 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4d8895-f14f-483a-8f96-ad0dc64e563b admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:54.167408 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4d8895-f14f-483a-8f96-ad0dc64e563b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:54.175776 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4d8895-f14f-483a-8f96-ad0dc64e563b admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:54.218508 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1124/2213] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:54 2022] GET /identity/v3/roles => generated 1420 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:41:54.223575 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739beb7c-243f-4125-addb-5e91c6f9999d admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:54.272556 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739beb7c-243f-4125-addb-5e91c6f9999d admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:54.278523 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdd2129-1188-49cc-b408-1a48952b7594 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:54.279331 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdd2129-1188-49cc-b408-1a48952b7594 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:54.279631 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdd2129-1188-49cc-b408-1a48952b7594 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:54.279918 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdd2129-1188-49cc-b408-1a48952b7594 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:54.280688 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bdd2129-1188-49cc-b408-1a48952b7594 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:54.291387 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bdd2129-1188-49cc-b408-1a48952b7594 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:54.318527 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1090/2214] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:54 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:41:54.323403 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4d8895-f14f-483a-8f96-ad0dc64e563b admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:54.369795 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4d8895-f14f-483a-8f96-ad0dc64e563b admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:54.402682 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c77ae8e-df91-4506-9ebe-3d1d2981b8d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:54.404253 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c77ae8e-df91-4506-9ebe-3d1d2981b8d4 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:54.404750 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c77ae8e-df91-4506-9ebe-3d1d2981b8d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:54.405228 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c77ae8e-df91-4506-9ebe-3d1d2981b8d4 admin admin] PATH_INFO: `/v3/projects/806e940a0a0d4f7d8a26525334b607b6/users/663fc2bca936427a8089f97a028dfef3/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:54.464246 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c77ae8e-df91-4506-9ebe-3d1d2981b8d4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=806e940a0a0d4f7d8a26525334b607b6, user_id=663fc2bca936427a8089f97a028dfef3, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:54.475599 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c77ae8e-df91-4506-9ebe-3d1d2981b8d4 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:54.579713 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1125/2215] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:41:54 2022] PUT /identity/v3/projects/806e940a0a0d4f7d8a26525334b607b6/users/663fc2bca936427a8089f97a028dfef3/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 257 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:41:54.595088 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77e4965-c97c-47a5-84a2-d72d394d5fe6 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:54.595088 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77e4965-c97c-47a5-84a2-d72d394d5fe6 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:54.595088 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77e4965-c97c-47a5-84a2-d72d394d5fe6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:54.664834 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-d77e4965-c97c-47a5-84a2-d72d394d5fe6 None None] MFA Rules not processed for user `663fc2bca936427a8089f97a028dfef3`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:54.668753 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-d77e4965-c97c-47a5-84a2-d72d394d5fe6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:54.903903 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1091/2216] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:54 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 311 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:41:54.936067 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c77ae8e-df91-4506-9ebe-3d1d2981b8d4 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:55.197330 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c77ae8e-df91-4506-9ebe-3d1d2981b8d4 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:55.238503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542a901c-e7e1-4c11-b37f-299f570fe25f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:55.239483 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542a901c-e7e1-4c11-b37f-299f570fe25f admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:55.239962 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542a901c-e7e1-4c11-b37f-299f570fe25f admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:55.240415 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542a901c-e7e1-4c11-b37f-299f570fe25f admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:55.241312 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-542a901c-e7e1-4c11-b37f-299f570fe25f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:55.243974 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-542a901c-e7e1-4c11-b37f-299f570fe25f admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:55.697121 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1126/2217] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:41:54 2022] POST /identity/v3/projects => generated 375 bytes in 762 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:41:55.702246 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77e4965-c97c-47a5-84a2-d72d394d5fe6 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:55.719289 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77e4965-c97c-47a5-84a2-d72d394d5fe6 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:55.834048 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b932eb99-a473-45a0-ac60-48b0ca55997e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:55.835001 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b932eb99-a473-45a0-ac60-48b0ca55997e admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:55.835416 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b932eb99-a473-45a0-ac60-48b0ca55997e admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:55.835803 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b932eb99-a473-45a0-ac60-48b0ca55997e admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:55.836882 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b932eb99-a473-45a0-ac60-48b0ca55997e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:55.839058 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b932eb99-a473-45a0-ac60-48b0ca55997e admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:55.949374 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1092/2218] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:41:55 2022] POST /identity/v3/users => generated 294 bytes in 247 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:41:55.954498 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542a901c-e7e1-4c11-b37f-299f570fe25f admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:55.978233 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542a901c-e7e1-4c11-b37f-299f570fe25f admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:55.998831 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c326e0-7ca8-4fea-a85b-51724beee97e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:56.005438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c326e0-7ca8-4fea-a85b-51724beee97e admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:56.006763 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c326e0-7ca8-4fea-a85b-51724beee97e admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:56.009461 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c326e0-7ca8-4fea-a85b-51724beee97e admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:56.011901 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c326e0-7ca8-4fea-a85b-51724beee97e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:56.018183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c326e0-7ca8-4fea-a85b-51724beee97e admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:56.079678 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1127/2219] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:55 2022] GET /identity/v3/roles => generated 1420 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:41:56.083391 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b932eb99-a473-45a0-ac60-48b0ca55997e admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:56.090227 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:56.097186 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b932eb99-a473-45a0-ac60-48b0ca55997e admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:56.114656 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3680926a-eee4-49f1-ba6c-c01eccfde50d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:56.115839 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3680926a-eee4-49f1-ba6c-c01eccfde50d admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:56.116226 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3680926a-eee4-49f1-ba6c-c01eccfde50d admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:56.116613 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3680926a-eee4-49f1-ba6c-c01eccfde50d admin admin] PATH_INFO: `/v3/projects/d80fcbe6c19a434faea608ffc213a8a4/users/f2687b3b90a644aea2aaa9c96ec4ad35/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:56.159465 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3680926a-eee4-49f1-ba6c-c01eccfde50d admin admin] RBAC: Authorizing `identity:create_grant(project_id=d80fcbe6c19a434faea608ffc213a8a4, user_id=f2687b3b90a644aea2aaa9c96ec4ad35, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:56.163366 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3680926a-eee4-49f1-ba6c-c01eccfde50d admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:56.270468 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1093/2220] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:41:56 2022] PUT /identity/v3/projects/d80fcbe6c19a434faea608ffc213a8a4/users/f2687b3b90a644aea2aaa9c96ec4ad35/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:56.275165 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c326e0-7ca8-4fea-a85b-51724beee97e admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:56.290889 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c326e0-7ca8-4fea-a85b-51724beee97e admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:56.527445 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b1a9af-68bd-4d7b-ae0f-c341e8438c0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:56.528145 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b1a9af-68bd-4d7b-ae0f-c341e8438c0e admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:56.528552 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b1a9af-68bd-4d7b-ae0f-c341e8438c0e admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:56.528721 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b1a9af-68bd-4d7b-ae0f-c341e8438c0e admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:56.529529 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b1a9af-68bd-4d7b-ae0f-c341e8438c0e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:56.531351 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b1a9af-68bd-4d7b-ae0f-c341e8438c0e admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:56.533668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:41:56.568455 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1128/2221] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:41:56 2022] GET /identity/v3/roles => generated 1420 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:41:56.581679 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3680926a-eee4-49f1-ba6c-c01eccfde50d admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:56.617365 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3680926a-eee4-49f1-ba6c-c01eccfde50d admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:56.647068 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3011b5d0-cf76-43f8-ae35-4ed2d1636da7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:56.648800 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3011b5d0-cf76-43f8-ae35-4ed2d1636da7 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:56.652322 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3011b5d0-cf76-43f8-ae35-4ed2d1636da7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:56.652738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3011b5d0-cf76-43f8-ae35-4ed2d1636da7 admin admin] PATH_INFO: `/v3/projects/d80fcbe6c19a434faea608ffc213a8a4/users/f2687b3b90a644aea2aaa9c96ec4ad35/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:56.663863 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3011b5d0-cf76-43f8-ae35-4ed2d1636da7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d80fcbe6c19a434faea608ffc213a8a4, user_id=f2687b3b90a644aea2aaa9c96ec4ad35, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:56.674533 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3011b5d0-cf76-43f8-ae35-4ed2d1636da7 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:56.694423 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1094/2222] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:41:56 2022] PUT /identity/v3/projects/d80fcbe6c19a434faea608ffc213a8a4/users/f2687b3b90a644aea2aaa9c96ec4ad35/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:41:56.707473 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7741bb3b-4a2a-46e6-bca1-4e73d89057bb None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:56.708104 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7741bb3b-4a2a-46e6-bca1-4e73d89057bb None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:56.708640 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7741bb3b-4a2a-46e6-bca1-4e73d89057bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:56.782889 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-7741bb3b-4a2a-46e6-bca1-4e73d89057bb None None] MFA Rules not processed for user `f2687b3b90a644aea2aaa9c96ec4ad35`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:41:56.785081 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-7741bb3b-4a2a-46e6-bca1-4e73d89057bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:56.978791 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1129/2223] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:41:56 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 272 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:41:56.986333 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3011b5d0-cf76-43f8-ae35-4ed2d1636da7 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:41:56.988346 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-3011b5d0-cf76-43f8-ae35-4ed2d1636da7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:41:57.189828 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3011b5d0-cf76-43f8-ae35-4ed2d1636da7 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:41:57.216224 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf52c50-8515-4272-9be2-c3c72b7070f7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:41:57.217189 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf52c50-8515-4272-9be2-c3c72b7070f7 service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:41:57.217419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf52c50-8515-4272-9be2-c3c72b7070f7 service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:41:57.217706 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf52c50-8515-4272-9be2-c3c72b7070f7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:41:57.454998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf52c50-8515-4272-9be2-c3c72b7070f7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:41:57.456503 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf52c50-8515-4272-9be2-c3c72b7070f7 service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:41:57.488731 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1095/2224] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:41:56 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 503 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:42:00.348665 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7741bb3b-4a2a-46e6-bca1-4e73d89057bb None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:00.365536 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7741bb3b-4a2a-46e6-bca1-4e73d89057bb None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:00.379945 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0655433-1a27-41af-8ba3-41cd39bb7320 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:00.380633 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0655433-1a27-41af-8ba3-41cd39bb7320 service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:00.380922 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0655433-1a27-41af-8ba3-41cd39bb7320 service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:00.383143 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0655433-1a27-41af-8ba3-41cd39bb7320 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:00.458494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0655433-1a27-41af-8ba3-41cd39bb7320 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:00.460619 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0655433-1a27-41af-8ba3-41cd39bb7320 service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:00.488535 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1130/2225] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:42:00 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:42:01.255031 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27ed324-5ae7-4c35-a8f8-82d675cc868e None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:01.255779 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27ed324-5ae7-4c35-a8f8-82d675cc868e None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:01.256282 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27ed324-5ae7-4c35-a8f8-82d675cc868e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:01.326978 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-f27ed324-5ae7-4c35-a8f8-82d675cc868e None None] MFA Rules not processed for user `3ac0d5d8de4943a3888abee3b24b8a6f`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:42:01.330275 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-f27ed324-5ae7-4c35-a8f8-82d675cc868e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:01.494981 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1096/2226] 10.222.0.16 () {58 vars in 967 bytes} [Tue Jul 12 17:42:01 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 246 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:42:01.512568 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0655433-1a27-41af-8ba3-41cd39bb7320 service cinder] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:01.514860 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-f0655433-1a27-41af-8ba3-41cd39bb7320 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:01.700209 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0655433-1a27-41af-8ba3-41cd39bb7320 service cinder] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:01.723036 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b268bf-d6bf-4cde-847c-6ff93fc23e81 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7906d1a19c642bd93507a27bf3e3dae', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:01.723720 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b268bf-d6bf-4cde-847c-6ff93fc23e81 service swift] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:01.724018 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b268bf-d6bf-4cde-847c-6ff93fc23e81 service swift] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:01.724795 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b268bf-d6bf-4cde-847c-6ff93fc23e81 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:01.749132 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b268bf-d6bf-4cde-847c-6ff93fc23e81 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:01.750955 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b268bf-d6bf-4cde-847c-6ff93fc23e81 service swift] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:01.773253 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1131/2227] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Jul 12 17:42:01 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 262 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 12 17:42:02.968500 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f27ed324-5ae7-4c35-a8f8-82d675cc868e None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:02.984475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f27ed324-5ae7-4c35-a8f8-82d675cc868e None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:02.991521 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54973b66-9a63-4ddd-8ce7-5d1d9b406fd2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:02.992522 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54973b66-9a63-4ddd-8ce7-5d1d9b406fd2 service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:02.993008 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54973b66-9a63-4ddd-8ce7-5d1d9b406fd2 service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:02.993488 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54973b66-9a63-4ddd-8ce7-5d1d9b406fd2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:03.009367 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54973b66-9a63-4ddd-8ce7-5d1d9b406fd2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:03.010909 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54973b66-9a63-4ddd-8ce7-5d1d9b406fd2 service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:03.024878 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1097/2228] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:42:02 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:42:04.551453 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37adaa2-1643-47b0-9368-76044d4b536e None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:04.552086 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37adaa2-1643-47b0-9368-76044d4b536e None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:04.552172 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37adaa2-1643-47b0-9368-76044d4b536e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:04.585671 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-c37adaa2-1643-47b0-9368-76044d4b536e None None] MFA Rules not processed for user `bde6ee9df10146d98991b0d4bb37fc48`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:42:04.592114 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-c37adaa2-1643-47b0-9368-76044d4b536e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:04.600694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:42:04.807391 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1132/2229] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:42:04 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 261 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:42:05.776462 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81f724f-62f9-4d6a-a437-3d56492255c5 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:05.777452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81f724f-62f9-4d6a-a437-3d56492255c5 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:05.777452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81f724f-62f9-4d6a-a437-3d56492255c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:05.894918 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:42:05.917291 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-c81f724f-62f9-4d6a-a437-3d56492255c5 None None] MFA Rules not processed for user `ed4726967dd844b692fa7badd510267a`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:42:05.919814 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-c81f724f-62f9-4d6a-a437-3d56492255c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:06.178701 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1098/2230] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:42:05 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 405 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:42:06.183720 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37adaa2-1643-47b0-9368-76044d4b536e None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:06.190758 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-c37adaa2-1643-47b0-9368-76044d4b536e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:06.451056 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37adaa2-1643-47b0-9368-76044d4b536e None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:06.460489 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5871ac74-a209-4162-84e2-483fc7ebe313 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:06.461572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5871ac74-a209-4162-84e2-483fc7ebe313 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:06.461902 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5871ac74-a209-4162-84e2-483fc7ebe313 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:06.462310 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5871ac74-a209-4162-84e2-483fc7ebe313 admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:06.482866 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5871ac74-a209-4162-84e2-483fc7ebe313 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:06.508359 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5871ac74-a209-4162-84e2-483fc7ebe313 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:06.647025 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1133/2231] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:42:06 2022] POST /identity/v3/projects => generated 385 bytes in 464 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:42:06.649042 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c81f724f-62f9-4d6a-a437-3d56492255c5 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:06.686522 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c81f724f-62f9-4d6a-a437-3d56492255c5 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:06.886006 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56fe02e9-62da-4a0a-88ea-1155dd672460 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:06.887736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fe02e9-62da-4a0a-88ea-1155dd672460 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:06.888247 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fe02e9-62da-4a0a-88ea-1155dd672460 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:06.888675 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fe02e9-62da-4a0a-88ea-1155dd672460 admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:06.890806 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56fe02e9-62da-4a0a-88ea-1155dd672460 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:06.895679 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56fe02e9-62da-4a0a-88ea-1155dd672460 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:07.009006 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1099/2232] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:42:06 2022] POST /identity/v3/users => generated 299 bytes in 361 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:42:07.013437 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5871ac74-a209-4162-84e2-483fc7ebe313 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:07.029283 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5871ac74-a209-4162-84e2-483fc7ebe313 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:07.053870 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fc2f2f-6864-4c13-b4f9-a74ea844dc9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:07.054603 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fc2f2f-6864-4c13-b4f9-a74ea844dc9e admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:07.054893 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fc2f2f-6864-4c13-b4f9-a74ea844dc9e admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:07.055154 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7fc2f2f-6864-4c13-b4f9-a74ea844dc9e admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:07.055927 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7fc2f2f-6864-4c13-b4f9-a74ea844dc9e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:07.057590 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7fc2f2f-6864-4c13-b4f9-a74ea844dc9e admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:07.098515 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56fe02e9-62da-4a0a-88ea-1155dd672460 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:07.116104 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56fe02e9-62da-4a0a-88ea-1155dd672460 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:07.117409 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1134/2233] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:42:07 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:42:07.134717 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fc2f2f-6864-4c13-b4f9-a74ea844dc9e admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:07.137390 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfda43f-a6da-48bc-ba84-8d59207e1220 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:07.138322 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfda43f-a6da-48bc-ba84-8d59207e1220 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:07.138641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfda43f-a6da-48bc-ba84-8d59207e1220 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:07.138925 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfda43f-a6da-48bc-ba84-8d59207e1220 admin admin] PATH_INFO: `/v3/projects/43ac40785a3e4fd7990f9edfd11b2f12/users/255cc0e9d9f748a19f8e104843259117/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:07.140882 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-f7fc2f2f-6864-4c13-b4f9-a74ea844dc9e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:07.206837 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfda43f-a6da-48bc-ba84-8d59207e1220 admin admin] RBAC: Authorizing `identity:create_grant(project_id=43ac40785a3e4fd7990f9edfd11b2f12, user_id=255cc0e9d9f748a19f8e104843259117, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:07.209624 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfda43f-a6da-48bc-ba84-8d59207e1220 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:07.301017 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1100/2234] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:42:07 2022] PUT /identity/v3/projects/43ac40785a3e4fd7990f9edfd11b2f12/users/255cc0e9d9f748a19f8e104843259117/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:42:07.302101 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfda43f-a6da-48bc-ba84-8d59207e1220 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:07.416912 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7fc2f2f-6864-4c13-b4f9-a74ea844dc9e admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:07.572430 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:42:07.620295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfda43f-a6da-48bc-ba84-8d59207e1220 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:07.648361 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26fd9fa3-2481-48b9-a411-e8474def5265 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:07.649064 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fd9fa3-2481-48b9-a411-e8474def5265 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:07.649794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fd9fa3-2481-48b9-a411-e8474def5265 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:07.649794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fd9fa3-2481-48b9-a411-e8474def5265 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:07.650378 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26fd9fa3-2481-48b9-a411-e8474def5265 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:07.652082 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26fd9fa3-2481-48b9-a411-e8474def5265 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:07.698274 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1101/2235] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:42:07 2022] GET /identity/v3/roles => generated 1420 bytes in 397 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:42:07.715666 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26fd9fa3-2481-48b9-a411-e8474def5265 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:07.721267 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eedcf00-0444-46ae-8120-333c0e9796dd service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f623be28c83247c78d3ea6873c347fe0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:07.726236 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:42:07.728687 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eedcf00-0444-46ae-8120-333c0e9796dd service placement] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:07.729057 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eedcf00-0444-46ae-8120-333c0e9796dd service placement] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:07.729312 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eedcf00-0444-46ae-8120-333c0e9796dd service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:07.737700 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-7eedcf00-0444-46ae-8120-333c0e9796dd service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:07.760778 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26fd9fa3-2481-48b9-a411-e8474def5265 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:07.788110 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9808b82-f4e7-43f9-9a80-7c43bd58edc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:07.794663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9808b82-f4e7-43f9-9a80-7c43bd58edc5 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:07.795175 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9808b82-f4e7-43f9-9a80-7c43bd58edc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:07.795693 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9808b82-f4e7-43f9-9a80-7c43bd58edc5 admin admin] PATH_INFO: `/v3/projects/43ac40785a3e4fd7990f9edfd11b2f12/users/255cc0e9d9f748a19f8e104843259117/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:07.799662 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eedcf00-0444-46ae-8120-333c0e9796dd service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:07.802613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eedcf00-0444-46ae-8120-333c0e9796dd service placement] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:07.818004 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9808b82-f4e7-43f9-9a80-7c43bd58edc5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=43ac40785a3e4fd7990f9edfd11b2f12, user_id=255cc0e9d9f748a19f8e104843259117, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:07.822907 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9808b82-f4e7-43f9-9a80-7c43bd58edc5 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:07.903687 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1135/2236] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:42:07 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 766 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:42:07.916681 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1102/2237] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:42:07 2022] PUT /identity/v3/projects/43ac40785a3e4fd7990f9edfd11b2f12/users/255cc0e9d9f748a19f8e104843259117/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:42:08.582810 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eedcf00-0444-46ae-8120-333c0e9796dd service placement] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:08.585022 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-7eedcf00-0444-46ae-8120-333c0e9796dd service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:08.644382 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eedcf00-0444-46ae-8120-333c0e9796dd service placement] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:08.799678 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae15c403-0f65-49cf-9306-102826ffd440 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:08.800871 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae15c403-0f65-49cf-9306-102826ffd440 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:08.801368 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae15c403-0f65-49cf-9306-102826ffd440 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:08.801802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae15c403-0f65-49cf-9306-102826ffd440 admin admin] PATH_INFO: `/v3/users/2c4deca7942842759f795ab5299b44a3` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:08.803838 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae15c403-0f65-49cf-9306-102826ffd440 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2c4deca7942842759f795ab5299b44a3)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:08.806546 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae15c403-0f65-49cf-9306-102826ffd440 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:09.113852 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ae15c403-0f65-49cf-9306-102826ffd440 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2c4deca7942842759f795ab5299b44a3'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:09.115721 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ae15c403-0f65-49cf-9306-102826ffd440 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2c4deca7942842759f795ab5299b44a3'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:09.118229 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ae15c403-0f65-49cf-9306-102826ffd440 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2c4deca7942842759f795ab5299b44a3'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:09.174873 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ae15c403-0f65-49cf-9306-102826ffd440 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2c4deca7942842759f795ab5299b44a3'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:09.262403 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ae15c403-0f65-49cf-9306-102826ffd440 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2c4deca7942842759f795ab5299b44a3'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:09.367206 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1136/2238] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:42:08 2022] DELETE /identity/v3/users/2c4deca7942842759f795ab5299b44a3 => generated 0 bytes in 785 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:42:09.643792 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9808b82-f4e7-43f9-9a80-7c43bd58edc5 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:09.646799 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-d9808b82-f4e7-43f9-9a80-7c43bd58edc5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:09.672596 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9808b82-f4e7-43f9-9a80-7c43bd58edc5 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:09.790740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e10acc3-3bbe-42f6-86ee-fb1fc0a38acd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:09.791767 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e10acc3-3bbe-42f6-86ee-fb1fc0a38acd admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:09.792156 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e10acc3-3bbe-42f6-86ee-fb1fc0a38acd admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:09.792454 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e10acc3-3bbe-42f6-86ee-fb1fc0a38acd admin admin] PATH_INFO: `/v3/users/74cfb3971bab4c0bb8eff71f13d0e1d8` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:09.793788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e10acc3-3bbe-42f6-86ee-fb1fc0a38acd admin admin] RBAC: Authorizing `identity:delete_user(user_id=74cfb3971bab4c0bb8eff71f13d0e1d8)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:09.796253 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e10acc3-3bbe-42f6-86ee-fb1fc0a38acd admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:09.951917 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae15c403-0f65-49cf-9306-102826ffd440 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:09.953486 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-ae15c403-0f65-49cf-9306-102826ffd440 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:09.990831 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae15c403-0f65-49cf-9306-102826ffd440 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:10.005649 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c3f388-72de-425d-9283-f5ca57825f91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:10.006869 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c3f388-72de-425d-9283-f5ca57825f91 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:10.007751 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c3f388-72de-425d-9283-f5ca57825f91 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:10.008207 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c3f388-72de-425d-9283-f5ca57825f91 admin admin] PATH_INFO: `/v3/projects/b3cb433fc8c44c87a642aaa9011ff624` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:10.010092 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c3f388-72de-425d-9283-f5ca57825f91 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b3cb433fc8c44c87a642aaa9011ff624)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:10.019981 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c3f388-72de-425d-9283-f5ca57825f91 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:10.058074 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-0e10acc3-3bbe-42f6-86ee-fb1fc0a38acd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '74cfb3971bab4c0bb8eff71f13d0e1d8'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:10.187738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-0e10acc3-3bbe-42f6-86ee-fb1fc0a38acd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '74cfb3971bab4c0bb8eff71f13d0e1d8'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:10.188445 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-0e10acc3-3bbe-42f6-86ee-fb1fc0a38acd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '74cfb3971bab4c0bb8eff71f13d0e1d8'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:10.201447 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-0e10acc3-3bbe-42f6-86ee-fb1fc0a38acd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '74cfb3971bab4c0bb8eff71f13d0e1d8'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:10.232539 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-0e10acc3-3bbe-42f6-86ee-fb1fc0a38acd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '74cfb3971bab4c0bb8eff71f13d0e1d8'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:10.251964 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1103/2239] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:42:09 2022] DELETE /identity/v3/users/74cfb3971bab4c0bb8eff71f13d0e1d8 => generated 0 bytes in 605 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:42:10.253570 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4566bb-d0d5-4c4e-ac03-ae70031f95b3 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:10.253876 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4566bb-d0d5-4c4e-ac03-ae70031f95b3 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:10.259774 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4566bb-d0d5-4c4e-ac03-ae70031f95b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:10.298670 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-cc4566bb-d0d5-4c4e-ac03-ae70031f95b3 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:42:10.301519 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-cc4566bb-d0d5-4c4e-ac03-ae70031f95b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:10.339070 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:42:10.369133 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-02c3f388-72de-425d-9283-f5ca57825f91 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b3cb433fc8c44c87a642aaa9011ff624', 'request_id': 'req-02c3f388-72de-425d-9283-f5ca57825f91'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:10.389710 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-02c3f388-72de-425d-9283-f5ca57825f91 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b3cb433fc8c44c87a642aaa9011ff624', 'request_id': 'req-02c3f388-72de-425d-9283-f5ca57825f91'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:10.436543 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1104/2240] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:42:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:42:10.447995 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc4566bb-d0d5-4c4e-ac03-ae70031f95b3 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:10.451819 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-cc4566bb-d0d5-4c4e-ac03-ae70031f95b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:10.477834 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:42:10.554697 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-02c3f388-72de-425d-9283-f5ca57825f91 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b3cb433fc8c44c87a642aaa9011ff624', 'request_id': 'req-02c3f388-72de-425d-9283-f5ca57825f91'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:10.564343 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc4566bb-d0d5-4c4e-ac03-ae70031f95b3 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:10.586188 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-02c3f388-72de-425d-9283-f5ca57825f91 admin admin] The token cache is being invalidate because project b3cb433fc8c44c87a642aaa9011ff624 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:42:10.586188 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-02c3f388-72de-425d-9283-f5ca57825f91 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:10.586188 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-02c3f388-72de-425d-9283-f5ca57825f91 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:10.592742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1137/2241] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:42:09 2022] DELETE /identity/v3/projects/b3cb433fc8c44c87a642aaa9011ff624 => generated 0 bytes in 637 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:42:10.594904 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5824645f-b166-4038-a54a-e576f6e96081 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:10.595206 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5824645f-b166-4038-a54a-e576f6e96081 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:10.595497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5824645f-b166-4038-a54a-e576f6e96081 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:10.658778 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-5824645f-b166-4038-a54a-e576f6e96081 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:42:10.660250 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-5824645f-b166-4038-a54a-e576f6e96081 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:10.665337 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb2a78f-8376-408f-a804-69d2f793edd6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:10.666940 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb2a78f-8376-408f-a804-69d2f793edd6 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:10.667651 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb2a78f-8376-408f-a804-69d2f793edd6 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:10.668097 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb2a78f-8376-408f-a804-69d2f793edd6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:10.680334 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-9bb2a78f-8376-408f-a804-69d2f793edd6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:10.798207 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb2a78f-8376-408f-a804-69d2f793edd6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:10.800116 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb2a78f-8376-408f-a804-69d2f793edd6 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:10.825403 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1138/2242] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:42:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:42:10.827934 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1105/2243] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:42:10 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:42:10.843904 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c78d66-8a62-4dc0-8614-af25e28139e4 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:10.846642 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c78d66-8a62-4dc0-8614-af25e28139e4 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:10.847315 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47c78d66-8a62-4dc0-8614-af25e28139e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:10.897245 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-47c78d66-8a62-4dc0-8614-af25e28139e4 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:42:10.898601 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-47c78d66-8a62-4dc0-8614-af25e28139e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:10.925651 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1139/2244] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:42:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:42:10.927287 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb2a78f-8376-408f-a804-69d2f793edd6 service neutron] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:10.945875 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb2a78f-8376-408f-a804-69d2f793edd6 service neutron] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:10.955859 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca26924a-0d58-442d-b457-8944b7c00ee6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:10.956668 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca26924a-0d58-442d-b457-8944b7c00ee6 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:10.957028 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca26924a-0d58-442d-b457-8944b7c00ee6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:10.957396 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca26924a-0d58-442d-b457-8944b7c00ee6 admin admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:10.958236 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca26924a-0d58-442d-b457-8944b7c00ee6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:42:10.960155 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca26924a-0d58-442d-b457-8944b7c00ee6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:10.962354 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca26924a-0d58-442d-b457-8944b7c00ee6 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:11.002203 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1106/2245] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:42:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:42:11.006741 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c78d66-8a62-4dc0-8614-af25e28139e4 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:11.024256 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47c78d66-8a62-4dc0-8614-af25e28139e4 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:11.033759 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c003c0-4864-4cee-a633-c811c2859b16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:11.041994 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c003c0-4864-4cee-a633-c811c2859b16 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:11.042709 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c003c0-4864-4cee-a633-c811c2859b16 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:11.043396 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c003c0-4864-4cee-a633-c811c2859b16 admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:11.047258 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c003c0-4864-4cee-a633-c811c2859b16 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:11.061255 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c003c0-4864-4cee-a633-c811c2859b16 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:11.254413 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1140/2246] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:42:11 2022] POST /identity/v3/projects => generated 385 bytes in 248 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:42:11.255457 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca26924a-0d58-442d-b457-8944b7c00ee6 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:11.270321 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca26924a-0d58-442d-b457-8944b7c00ee6 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:11.529276 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25f2b09-eb7b-47bc-9689-b96b26813d4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:11.530594 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25f2b09-eb7b-47bc-9689-b96b26813d4b admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:11.531140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25f2b09-eb7b-47bc-9689-b96b26813d4b admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:11.531674 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25f2b09-eb7b-47bc-9689-b96b26813d4b admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:11.534140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25f2b09-eb7b-47bc-9689-b96b26813d4b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:11.538375 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25f2b09-eb7b-47bc-9689-b96b26813d4b admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:11.956837 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1107/2247] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:42:11 2022] POST /identity/v3/users => generated 293 bytes in 675 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:42:11.958571 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c003c0-4864-4cee-a633-c811c2859b16 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:11.958571 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c003c0-4864-4cee-a633-c811c2859b16 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:11.975878 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8863a5-6b3f-4187-891a-3e777448648d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:11.975878 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8863a5-6b3f-4187-891a-3e777448648d admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:11.975878 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8863a5-6b3f-4187-891a-3e777448648d admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:11.975878 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8863a5-6b3f-4187-891a-3e777448648d admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:11.976618 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8863a5-6b3f-4187-891a-3e777448648d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:11.984539 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8863a5-6b3f-4187-891a-3e777448648d admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:12.011388 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1141/2248] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:42:11 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 12 17:42:12.015195 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25f2b09-eb7b-47bc-9689-b96b26813d4b admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:12.036895 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25f2b09-eb7b-47bc-9689-b96b26813d4b admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:12.053151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf86f9b-1a19-4ad4-82b6-62cd899c0d29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:12.064881 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf86f9b-1a19-4ad4-82b6-62cd899c0d29 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:12.069802 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf86f9b-1a19-4ad4-82b6-62cd899c0d29 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:12.070332 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf86f9b-1a19-4ad4-82b6-62cd899c0d29 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:12.072429 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf86f9b-1a19-4ad4-82b6-62cd899c0d29 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:12.073779 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8863a5-6b3f-4187-891a-3e777448648d admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:12.076567 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf86f9b-1a19-4ad4-82b6-62cd899c0d29 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:12.081026 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-0f8863a5-6b3f-4187-891a-3e777448648d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:12.109273 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1108/2249] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:42:12 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:42:12.115933 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf86f9b-1a19-4ad4-82b6-62cd899c0d29 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:12.125296 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8863a5-6b3f-4187-891a-3e777448648d admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:12.135326 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf0fba9-7803-4b96-9d94-b243a0dfc7c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:12.137210 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf0fba9-7803-4b96-9d94-b243a0dfc7c1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:12.137681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf0fba9-7803-4b96-9d94-b243a0dfc7c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:12.138466 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf0fba9-7803-4b96-9d94-b243a0dfc7c1 admin admin] PATH_INFO: `/v3/projects/c0bbd2ab656e48188ef7b9362f01de15` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:12.140742 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf86f9b-1a19-4ad4-82b6-62cd899c0d29 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:12.141887 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf0fba9-7803-4b96-9d94-b243a0dfc7c1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c0bbd2ab656e48188ef7b9362f01de15)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:12.143963 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf0fba9-7803-4b96-9d94-b243a0dfc7c1 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:12.150746 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057c6c04-9f98-4939-8bc9-c1d55ff90f58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:12.152658 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057c6c04-9f98-4939-8bc9-c1d55ff90f58 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:12.153202 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057c6c04-9f98-4939-8bc9-c1d55ff90f58 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:12.154266 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057c6c04-9f98-4939-8bc9-c1d55ff90f58 admin admin] PATH_INFO: `/v3/projects/5e60ecb1b8df4812b631e18f424f07a3/users/993f75033aee4174a7460663016c3dd2/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:12.181138 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-057c6c04-9f98-4939-8bc9-c1d55ff90f58 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5e60ecb1b8df4812b631e18f424f07a3, user_id=993f75033aee4174a7460663016c3dd2, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:12.185319 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-057c6c04-9f98-4939-8bc9-c1d55ff90f58 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:12.382779 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1109/2250] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:42:12 2022] PUT /identity/v3/projects/5e60ecb1b8df4812b631e18f424f07a3/users/993f75033aee4174a7460663016c3dd2/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:42:12.397479 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b43355-3849-42ca-aa04-ded38a914f25 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:12.397860 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b43355-3849-42ca-aa04-ded38a914f25 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:12.398089 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b43355-3849-42ca-aa04-ded38a914f25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:12.476841 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-d0b43355-3849-42ca-aa04-ded38a914f25 None None] MFA Rules not processed for user `993f75033aee4174a7460663016c3dd2`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:42:12.477845 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-d0b43355-3849-42ca-aa04-ded38a914f25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:12.478959 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-2cf0fba9-7803-4b96-9d94-b243a0dfc7c1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c0bbd2ab656e48188ef7b9362f01de15', 'request_id': 'req-2cf0fba9-7803-4b96-9d94-b243a0dfc7c1'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:12.508144 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-2cf0fba9-7803-4b96-9d94-b243a0dfc7c1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c0bbd2ab656e48188ef7b9362f01de15', 'request_id': 'req-2cf0fba9-7803-4b96-9d94-b243a0dfc7c1'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:12.690410 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-2cf0fba9-7803-4b96-9d94-b243a0dfc7c1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c0bbd2ab656e48188ef7b9362f01de15', 'request_id': 'req-2cf0fba9-7803-4b96-9d94-b243a0dfc7c1'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:12.720220 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-2cf0fba9-7803-4b96-9d94-b243a0dfc7c1 admin admin] The token cache is being invalidate because project c0bbd2ab656e48188ef7b9362f01de15 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:42:12.724409 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-2cf0fba9-7803-4b96-9d94-b243a0dfc7c1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:12.724996 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-2cf0fba9-7803-4b96-9d94-b243a0dfc7c1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:12.728608 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1142/2251] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:42:12 2022] DELETE /identity/v3/projects/c0bbd2ab656e48188ef7b9362f01de15 => generated 0 bytes in 660 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:42:12.738106 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf0fba9-7803-4b96-9d94-b243a0dfc7c1 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:12.740462 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1110/2252] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:42:12 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 344 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:42:12.740669 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-2cf0fba9-7803-4b96-9d94-b243a0dfc7c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:12.761605 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b43355-3849-42ca-aa04-ded38a914f25 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:12.763354 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-d0b43355-3849-42ca-aa04-ded38a914f25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:12.981140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b43355-3849-42ca-aa04-ded38a914f25 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:12.993643 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033df9f6-21ec-4c74-9897-644e5b57e4d4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:12.994677 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033df9f6-21ec-4c74-9897-644e5b57e4d4 service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:12.995129 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033df9f6-21ec-4c74-9897-644e5b57e4d4 service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:12.995418 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033df9f6-21ec-4c74-9897-644e5b57e4d4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:12.999406 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-033df9f6-21ec-4c74-9897-644e5b57e4d4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:13.001317 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf0fba9-7803-4b96-9d94-b243a0dfc7c1 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:13.027259 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033df9f6-21ec-4c74-9897-644e5b57e4d4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:13.032601 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033df9f6-21ec-4c74-9897-644e5b57e4d4 service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:13.042057 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e33860-01cc-466e-bf24-6763626ac598 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:13.043613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e33860-01cc-466e-bf24-6763626ac598 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:13.044109 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e33860-01cc-466e-bf24-6763626ac598 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:13.044609 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e33860-01cc-466e-bf24-6763626ac598 admin admin] PATH_INFO: `/v3/users/663fc2bca936427a8089f97a028dfef3` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:13.050592 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e33860-01cc-466e-bf24-6763626ac598 admin admin] RBAC: Authorizing `identity:delete_user(user_id=663fc2bca936427a8089f97a028dfef3)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:13.062858 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e33860-01cc-466e-bf24-6763626ac598 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:13.067965 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1111/2253] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:42:12 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:42:13.393486 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-03e33860-01cc-466e-bf24-6763626ac598 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '663fc2bca936427a8089f97a028dfef3'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:13.394529 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-03e33860-01cc-466e-bf24-6763626ac598 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '663fc2bca936427a8089f97a028dfef3'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:13.400204 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-03e33860-01cc-466e-bf24-6763626ac598 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '663fc2bca936427a8089f97a028dfef3'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:13.439802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-03e33860-01cc-466e-bf24-6763626ac598 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '663fc2bca936427a8089f97a028dfef3'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:13.486782 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:DELETE=12,keystone:SELECT=85,keystone:INSERT=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:42:13.501989 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-03e33860-01cc-466e-bf24-6763626ac598 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '663fc2bca936427a8089f97a028dfef3'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:13.646480 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1143/2254] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:42:12 2022] DELETE /identity/v3/users/663fc2bca936427a8089f97a028dfef3 => generated 0 bytes in 909 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:42:14.325389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:42:15.317617 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033df9f6-21ec-4c74-9897-644e5b57e4d4 service cinder] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:15.337184 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033df9f6-21ec-4c74-9897-644e5b57e4d4 service cinder] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:15.485742 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42770c7-4a82-4ded-be9f-fc534a7ebfb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:15.487100 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42770c7-4a82-4ded-be9f-fc534a7ebfb8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:15.488289 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42770c7-4a82-4ded-be9f-fc534a7ebfb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:15.488758 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42770c7-4a82-4ded-be9f-fc534a7ebfb8 admin admin] PATH_INFO: `/v3/projects/806e940a0a0d4f7d8a26525334b607b6` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:15.490528 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42770c7-4a82-4ded-be9f-fc534a7ebfb8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=806e940a0a0d4f7d8a26525334b607b6)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:15.492929 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42770c7-4a82-4ded-be9f-fc534a7ebfb8 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:15.699764 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-f42770c7-4a82-4ded-be9f-fc534a7ebfb8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '806e940a0a0d4f7d8a26525334b607b6', 'request_id': 'req-f42770c7-4a82-4ded-be9f-fc534a7ebfb8'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:15.721536 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-f42770c7-4a82-4ded-be9f-fc534a7ebfb8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '806e940a0a0d4f7d8a26525334b607b6', 'request_id': 'req-f42770c7-4a82-4ded-be9f-fc534a7ebfb8'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:15.730578 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-f42770c7-4a82-4ded-be9f-fc534a7ebfb8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '806e940a0a0d4f7d8a26525334b607b6', 'request_id': 'req-f42770c7-4a82-4ded-be9f-fc534a7ebfb8'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:15.885975 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-f42770c7-4a82-4ded-be9f-fc534a7ebfb8 admin admin] The token cache is being invalidate because project 806e940a0a0d4f7d8a26525334b607b6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:42:15.886475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-f42770c7-4a82-4ded-be9f-fc534a7ebfb8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:15.890154 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-f42770c7-4a82-4ded-be9f-fc534a7ebfb8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:15.900674 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1112/2255] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:42:15 2022] DELETE /identity/v3/projects/806e940a0a0d4f7d8a26525334b607b6 => generated 0 bytes in 580 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:42:15.902501 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e33860-01cc-466e-bf24-6763626ac598 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:15.903982 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-03e33860-01cc-466e-bf24-6763626ac598 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:15.919375 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e33860-01cc-466e-bf24-6763626ac598 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:16.017215 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6510c6cb-77af-426a-b011-88e36988405d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:16.018622 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6510c6cb-77af-426a-b011-88e36988405d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:16.019227 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6510c6cb-77af-426a-b011-88e36988405d admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:16.027165 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6510c6cb-77af-426a-b011-88e36988405d admin admin] PATH_INFO: `/v3/users/f2687b3b90a644aea2aaa9c96ec4ad35` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:16.034544 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6510c6cb-77af-426a-b011-88e36988405d admin admin] RBAC: Authorizing `identity:delete_user(user_id=f2687b3b90a644aea2aaa9c96ec4ad35)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:16.041141 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6510c6cb-77af-426a-b011-88e36988405d admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:16.365976 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:42:16.405796 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-6510c6cb-77af-426a-b011-88e36988405d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f2687b3b90a644aea2aaa9c96ec4ad35'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:16.405796 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-6510c6cb-77af-426a-b011-88e36988405d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f2687b3b90a644aea2aaa9c96ec4ad35'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:16.406842 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-6510c6cb-77af-426a-b011-88e36988405d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f2687b3b90a644aea2aaa9c96ec4ad35'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:16.421429 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-6510c6cb-77af-426a-b011-88e36988405d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f2687b3b90a644aea2aaa9c96ec4ad35'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:16.479563 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-6510c6cb-77af-426a-b011-88e36988405d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f2687b3b90a644aea2aaa9c96ec4ad35'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:16.608137 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1144/2256] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:42:15 2022] DELETE /identity/v3/users/f2687b3b90a644aea2aaa9c96ec4ad35 => generated 0 bytes in 701 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:42:16.685758 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:42:17.495454 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42770c7-4a82-4ded-be9f-fc534a7ebfb8 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:17.497874 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-f42770c7-4a82-4ded-be9f-fc534a7ebfb8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:17.520880 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42770c7-4a82-4ded-be9f-fc534a7ebfb8 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:17.615857 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc64de0-d295-491b-85f2-e6f9e19b7fc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:17.617432 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc64de0-d295-491b-85f2-e6f9e19b7fc0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:17.617942 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc64de0-d295-491b-85f2-e6f9e19b7fc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:17.618488 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc64de0-d295-491b-85f2-e6f9e19b7fc0 admin admin] PATH_INFO: `/v3/users/993f75033aee4174a7460663016c3dd2` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:17.621309 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc64de0-d295-491b-85f2-e6f9e19b7fc0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=993f75033aee4174a7460663016c3dd2)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:17.624679 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc64de0-d295-491b-85f2-e6f9e19b7fc0 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:18.017266 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-9fc64de0-d295-491b-85f2-e6f9e19b7fc0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '993f75033aee4174a7460663016c3dd2'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:18.102557 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-9fc64de0-d295-491b-85f2-e6f9e19b7fc0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '993f75033aee4174a7460663016c3dd2'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:18.104003 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-9fc64de0-d295-491b-85f2-e6f9e19b7fc0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '993f75033aee4174a7460663016c3dd2'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:18.118036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-9fc64de0-d295-491b-85f2-e6f9e19b7fc0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '993f75033aee4174a7460663016c3dd2'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:18.155661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-9fc64de0-d295-491b-85f2-e6f9e19b7fc0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '993f75033aee4174a7460663016c3dd2'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:18.178728 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1113/2257] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:42:17 2022] DELETE /identity/v3/users/993f75033aee4174a7460663016c3dd2 => generated 0 bytes in 684 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:42:18.185521 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6510c6cb-77af-426a-b011-88e36988405d admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:18.186382 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3145de93-501a-4a3f-bedb-27f91afb75d2 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:18.186961 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3145de93-501a-4a3f-bedb-27f91afb75d2 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:18.187427 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3145de93-501a-4a3f-bedb-27f91afb75d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:18.189156 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-6510c6cb-77af-426a-b011-88e36988405d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:18.212095 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-3145de93-501a-4a3f-bedb-27f91afb75d2 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:42:18.213886 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-3145de93-501a-4a3f-bedb-27f91afb75d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:18.218310 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6510c6cb-77af-426a-b011-88e36988405d admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:18.296674 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:42:18.342658 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1114/2258] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:42:18 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:42:18.365850 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3145de93-501a-4a3f-bedb-27f91afb75d2 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:18.371848 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-3145de93-501a-4a3f-bedb-27f91afb75d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:18.385657 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa41deb-57f1-4955-9983-b07de67571fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:18.387063 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa41deb-57f1-4955-9983-b07de67571fe admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:18.387552 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa41deb-57f1-4955-9983-b07de67571fe admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:18.388204 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa41deb-57f1-4955-9983-b07de67571fe admin admin] PATH_INFO: `/v3/projects/d80fcbe6c19a434faea608ffc213a8a4` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:18.395456 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa41deb-57f1-4955-9983-b07de67571fe admin admin] RBAC: Authorizing `identity:delete_project(project_id=d80fcbe6c19a434faea608ffc213a8a4)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:18.397644 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa41deb-57f1-4955-9983-b07de67571fe admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:18.510394 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3145de93-501a-4a3f-bedb-27f91afb75d2 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:18.527277 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-3aa41deb-57f1-4955-9983-b07de67571fe admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd80fcbe6c19a434faea608ffc213a8a4', 'request_id': 'req-3aa41deb-57f1-4955-9983-b07de67571fe'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:18.537454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-3aa41deb-57f1-4955-9983-b07de67571fe admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd80fcbe6c19a434faea608ffc213a8a4', 'request_id': 'req-3aa41deb-57f1-4955-9983-b07de67571fe'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:18.594744 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c166377e-c57b-41da-a5a7-a3a748ce4d10 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:18.595473 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c166377e-c57b-41da-a5a7-a3a748ce4d10 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:18.595884 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c166377e-c57b-41da-a5a7-a3a748ce4d10 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:18.596025 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c166377e-c57b-41da-a5a7-a3a748ce4d10 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:18.652230 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-3aa41deb-57f1-4955-9983-b07de67571fe admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd80fcbe6c19a434faea608ffc213a8a4', 'request_id': 'req-3aa41deb-57f1-4955-9983-b07de67571fe'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:18.673420 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-3aa41deb-57f1-4955-9983-b07de67571fe admin admin] The token cache is being invalidate because project d80fcbe6c19a434faea608ffc213a8a4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:42:18.673717 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-3aa41deb-57f1-4955-9983-b07de67571fe admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:18.674594 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-3aa41deb-57f1-4955-9983-b07de67571fe admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:18.681979 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1145/2259] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:42:18 2022] DELETE /identity/v3/projects/d80fcbe6c19a434faea608ffc213a8a4 => generated 0 bytes in 492 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:42:18.683724 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad44e22-7754-4a47-8c06-071a8bdde6e4 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:18.684066 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad44e22-7754-4a47-8c06-071a8bdde6e4 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:18.684422 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad44e22-7754-4a47-8c06-071a8bdde6e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:18.728722 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c166377e-c57b-41da-a5a7-a3a748ce4d10 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:18.730801 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-dad44e22-7754-4a47-8c06-071a8bdde6e4 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:42:18.732126 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-dad44e22-7754-4a47-8c06-071a8bdde6e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:18.732581 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c166377e-c57b-41da-a5a7-a3a748ce4d10 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:18.764742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1146/2260] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:42:18 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 72 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:42:18.765040 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1115/2261] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:42:18 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 395 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:42:18.768140 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c951350-2172-4cbd-b0de-ff5f9662c281 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:18.769431 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c951350-2172-4cbd-b0de-ff5f9662c281 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:18.769878 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c951350-2172-4cbd-b0de-ff5f9662c281 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:18.794235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-9c951350-2172-4cbd-b0de-ff5f9662c281 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:42:18.795945 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-9c951350-2172-4cbd-b0de-ff5f9662c281 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:18.811164 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1147/2262] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:42:18 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:42:18.814330 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c166377e-c57b-41da-a5a7-a3a748ce4d10 service neutron] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:18.815924 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-c166377e-c57b-41da-a5a7-a3a748ce4d10 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:18.849746 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c166377e-c57b-41da-a5a7-a3a748ce4d10 service neutron] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:18.862095 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d00f90-968f-4797-bec0-59a4ed81c9d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:18.862362 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d00f90-968f-4797-bec0-59a4ed81c9d6 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:18.862550 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d00f90-968f-4797-bec0-59a4ed81c9d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:18.862758 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d00f90-968f-4797-bec0-59a4ed81c9d6 admin admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:18.864152 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d00f90-968f-4797-bec0-59a4ed81c9d6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:42:18.864152 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d00f90-968f-4797-bec0-59a4ed81c9d6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:18.866220 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d00f90-968f-4797-bec0-59a4ed81c9d6 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:18.898392 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1116/2263] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:42:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:42:18.901028 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c951350-2172-4cbd-b0de-ff5f9662c281 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:18.917561 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c951350-2172-4cbd-b0de-ff5f9662c281 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:18.925171 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba75a2f0-949e-4911-9462-59db21b5133c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:18.925990 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba75a2f0-949e-4911-9462-59db21b5133c admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:18.926332 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba75a2f0-949e-4911-9462-59db21b5133c admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:18.926817 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba75a2f0-949e-4911-9462-59db21b5133c admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:18.935604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba75a2f0-949e-4911-9462-59db21b5133c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:18.941553 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba75a2f0-949e-4911-9462-59db21b5133c admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:19.075115 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:42:19.087725 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1148/2264] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:42:18 2022] POST /identity/v3/projects => generated 375 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:42:19.092277 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d00f90-968f-4797-bec0-59a4ed81c9d6 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:19.115096 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d00f90-968f-4797-bec0-59a4ed81c9d6 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:19.304677 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cba6e1-04fa-4526-a87e-446ee396c589 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:19.306406 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cba6e1-04fa-4526-a87e-446ee396c589 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:19.306406 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cba6e1-04fa-4526-a87e-446ee396c589 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:19.306406 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cba6e1-04fa-4526-a87e-446ee396c589 admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:19.306773 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65cba6e1-04fa-4526-a87e-446ee396c589 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:19.315360 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65cba6e1-04fa-4526-a87e-446ee396c589 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:19.493585 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1117/2265] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:42:19 2022] POST /identity/v3/users => generated 288 bytes in 402 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:42:19.502123 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba75a2f0-949e-4911-9462-59db21b5133c admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:19.537341 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba75a2f0-949e-4911-9462-59db21b5133c admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:19.571689 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0667a80-bad3-48e5-a0a1-6aee7d2dd8b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:19.574696 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0667a80-bad3-48e5-a0a1-6aee7d2dd8b5 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:19.575844 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0667a80-bad3-48e5-a0a1-6aee7d2dd8b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:19.577345 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0667a80-bad3-48e5-a0a1-6aee7d2dd8b5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:19.579229 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0667a80-bad3-48e5-a0a1-6aee7d2dd8b5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:19.591312 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0667a80-bad3-48e5-a0a1-6aee7d2dd8b5 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:19.611620 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:42:19.647140 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1149/2266] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:42:19 2022] GET /identity/v3/roles => generated 1420 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:42:19.652375 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cba6e1-04fa-4526-a87e-446ee396c589 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:19.695828 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cba6e1-04fa-4526-a87e-446ee396c589 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:19.721749 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c9827e2-53ba-422c-8f35-2503de155278 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:19.725506 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9827e2-53ba-422c-8f35-2503de155278 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:19.726464 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9827e2-53ba-422c-8f35-2503de155278 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:19.726883 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9827e2-53ba-422c-8f35-2503de155278 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:19.727733 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c9827e2-53ba-422c-8f35-2503de155278 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:19.737320 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c9827e2-53ba-422c-8f35-2503de155278 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:19.767959 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1118/2267] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:42:19 2022] GET /identity/v3/roles => generated 1420 bytes in 117 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 12 17:42:19.777744 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0667a80-bad3-48e5-a0a1-6aee7d2dd8b5 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:19.810189 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0667a80-bad3-48e5-a0a1-6aee7d2dd8b5 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:19.828180 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-008e4013-0cb1-4bbb-a608-ac20565234fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:19.830927 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008e4013-0cb1-4bbb-a608-ac20565234fb admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:19.831505 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008e4013-0cb1-4bbb-a608-ac20565234fb admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:19.832957 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-008e4013-0cb1-4bbb-a608-ac20565234fb admin admin] PATH_INFO: `/v3/projects/019d9f28339c4316bb3cda3e4c6ea908/users/7b076f0335184c22a2aa07bf484c631e/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:19.960003 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008e4013-0cb1-4bbb-a608-ac20565234fb admin admin] RBAC: Authorizing `identity:create_grant(project_id=019d9f28339c4316bb3cda3e4c6ea908, user_id=7b076f0335184c22a2aa07bf484c631e, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:19.965986 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-008e4013-0cb1-4bbb-a608-ac20565234fb admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:20.038501 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c9827e2-53ba-422c-8f35-2503de155278 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:20.043079 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-5c9827e2-53ba-422c-8f35-2503de155278 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:20.073407 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c9827e2-53ba-422c-8f35-2503de155278 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:20.085179 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb76cc6a-dcce-4034-8ec1-d72c01f627b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:20.087092 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb76cc6a-dcce-4034-8ec1-d72c01f627b1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:20.089412 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb76cc6a-dcce-4034-8ec1-d72c01f627b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:20.089849 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb76cc6a-dcce-4034-8ec1-d72c01f627b1 admin admin] PATH_INFO: `/v3/projects/5e60ecb1b8df4812b631e18f424f07a3` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:20.092563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb76cc6a-dcce-4034-8ec1-d72c01f627b1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5e60ecb1b8df4812b631e18f424f07a3)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:20.097132 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb76cc6a-dcce-4034-8ec1-d72c01f627b1 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:20.161288 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1150/2268] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:42:19 2022] PUT /identity/v3/projects/019d9f28339c4316bb3cda3e4c6ea908/users/7b076f0335184c22a2aa07bf484c631e/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 380 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:42:20.163115 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36cbc9e-b2fa-4528-99ca-0a1dee0feb13 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:20.163513 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36cbc9e-b2fa-4528-99ca-0a1dee0feb13 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:20.163846 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36cbc9e-b2fa-4528-99ca-0a1dee0feb13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:20.239161 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-a36cbc9e-b2fa-4528-99ca-0a1dee0feb13 None None] MFA Rules not processed for user `7b076f0335184c22a2aa07bf484c631e`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:42:20.240658 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-a36cbc9e-b2fa-4528-99ca-0a1dee0feb13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:20.331234 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-fb76cc6a-dcce-4034-8ec1-d72c01f627b1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5e60ecb1b8df4812b631e18f424f07a3', 'request_id': 'req-fb76cc6a-dcce-4034-8ec1-d72c01f627b1'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:20.342783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-fb76cc6a-dcce-4034-8ec1-d72c01f627b1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5e60ecb1b8df4812b631e18f424f07a3', 'request_id': 'req-fb76cc6a-dcce-4034-8ec1-d72c01f627b1'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:20.354155 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-fb76cc6a-dcce-4034-8ec1-d72c01f627b1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5e60ecb1b8df4812b631e18f424f07a3', 'request_id': 'req-fb76cc6a-dcce-4034-8ec1-d72c01f627b1'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:20.358966 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1151/2269] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:42:20 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:42:20.370922 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a36cbc9e-b2fa-4528-99ca-0a1dee0feb13 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:20.372690 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-a36cbc9e-b2fa-4528-99ca-0a1dee0feb13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:20.465831 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a36cbc9e-b2fa-4528-99ca-0a1dee0feb13 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:20.510363 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-fb76cc6a-dcce-4034-8ec1-d72c01f627b1 admin admin] The token cache is being invalidate because project 5e60ecb1b8df4812b631e18f424f07a3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:42:20.510363 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-fb76cc6a-dcce-4034-8ec1-d72c01f627b1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:20.511050 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-fb76cc6a-dcce-4034-8ec1-d72c01f627b1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:42:20.513809 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1119/2270] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:42:20 2022] DELETE /identity/v3/projects/5e60ecb1b8df4812b631e18f424f07a3 => generated 0 bytes in 477 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:42:20.521554 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3d682b-cd2e-4a6d-a020-e07ea6f46a2d None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:20.522079 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3d682b-cd2e-4a6d-a020-e07ea6f46a2d None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:20.522682 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3d682b-cd2e-4a6d-a020-e07ea6f46a2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:20.553470 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9071af-6d7b-4724-bfe9-1404eb21d1f0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:20.554224 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9071af-6d7b-4724-bfe9-1404eb21d1f0 service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:20.555284 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9071af-6d7b-4724-bfe9-1404eb21d1f0 service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:20.555692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9071af-6d7b-4724-bfe9-1404eb21d1f0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:20.558165 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-df9071af-6d7b-4724-bfe9-1404eb21d1f0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:20.573600 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-7e3d682b-cd2e-4a6d-a020-e07ea6f46a2d None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:42:20.576880 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-7e3d682b-cd2e-4a6d-a020-e07ea6f46a2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:20.659865 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:42:20.745246 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:42:20.759460 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df9071af-6d7b-4724-bfe9-1404eb21d1f0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:20.763421 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df9071af-6d7b-4724-bfe9-1404eb21d1f0 service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:20.773790 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1120/2271] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:42:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 248 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:42:20.777341 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0698b662-2983-47d5-910e-188124ff5de6 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:20.778202 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0698b662-2983-47d5-910e-188124ff5de6 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:20.780993 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0698b662-2983-47d5-910e-188124ff5de6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:20.789005 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1152/2272] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:42:20 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 419 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:42:20.843179 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-0698b662-2983-47d5-910e-188124ff5de6 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:42:20.847072 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-0698b662-2983-47d5-910e-188124ff5de6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:20.870637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1121/2273] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:42:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:42:20.873508 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9071af-6d7b-4724-bfe9-1404eb21d1f0 service cinder] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:20.897176 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9071af-6d7b-4724-bfe9-1404eb21d1f0 service cinder] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:20.917436 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1749a46f-e8af-4022-baf5-93be88de71cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:20.918752 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1749a46f-e8af-4022-baf5-93be88de71cc admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:20.919055 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1749a46f-e8af-4022-baf5-93be88de71cc admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:20.919352 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1749a46f-e8af-4022-baf5-93be88de71cc admin admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:20.920123 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1749a46f-e8af-4022-baf5-93be88de71cc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:42:20.920446 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1749a46f-e8af-4022-baf5-93be88de71cc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:20.930745 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1749a46f-e8af-4022-baf5-93be88de71cc admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:20.996792 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0698b662-2983-47d5-910e-188124ff5de6 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:20.997963 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1153/2274] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:42:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:42:21.031883 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0698b662-2983-47d5-910e-188124ff5de6 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:21.040571 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac4e580-4b8b-45ae-9096-f64425f2793c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:21.041382 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac4e580-4b8b-45ae-9096-f64425f2793c admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:21.041997 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac4e580-4b8b-45ae-9096-f64425f2793c admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:21.042298 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac4e580-4b8b-45ae-9096-f64425f2793c admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:21.043932 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac4e580-4b8b-45ae-9096-f64425f2793c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:21.046504 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ac4e580-4b8b-45ae-9096-f64425f2793c admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:21.423573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1122/2275] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:42:20 2022] POST /identity/v3/projects => generated 365 bytes in 424 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:42:21.424775 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1749a46f-e8af-4022-baf5-93be88de71cc admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:21.435752 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1749a46f-e8af-4022-baf5-93be88de71cc admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:21.522112 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4188c0f-fbbb-421d-a6fd-64440ac48357 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:21.523346 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4188c0f-fbbb-421d-a6fd-64440ac48357 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:21.523983 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4188c0f-fbbb-421d-a6fd-64440ac48357 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:21.524487 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4188c0f-fbbb-421d-a6fd-64440ac48357 admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:21.525410 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4188c0f-fbbb-421d-a6fd-64440ac48357 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:21.527659 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4188c0f-fbbb-421d-a6fd-64440ac48357 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:21.694482 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1154/2276] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:42:21 2022] POST /identity/v3/users => generated 283 bytes in 270 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:42:21.697589 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac4e580-4b8b-45ae-9096-f64425f2793c admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:21.710388 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac4e580-4b8b-45ae-9096-f64425f2793c admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:21.726368 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317f1d31-c9bf-4710-98c6-4e27f9597895 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:21.727706 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317f1d31-c9bf-4710-98c6-4e27f9597895 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:21.728207 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317f1d31-c9bf-4710-98c6-4e27f9597895 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:21.728805 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-317f1d31-c9bf-4710-98c6-4e27f9597895 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:21.731063 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-317f1d31-c9bf-4710-98c6-4e27f9597895 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:21.736162 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-317f1d31-c9bf-4710-98c6-4e27f9597895 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:21.772464 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4188c0f-fbbb-421d-a6fd-64440ac48357 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:21.773049 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1123/2277] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:42:21 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:42:21.796132 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4188c0f-fbbb-421d-a6fd-64440ac48357 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:21.807585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08abe406-48e9-4dad-b81b-7b99d8d38ec1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:21.809106 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08abe406-48e9-4dad-b81b-7b99d8d38ec1 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:21.809710 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08abe406-48e9-4dad-b81b-7b99d8d38ec1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:21.810294 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08abe406-48e9-4dad-b81b-7b99d8d38ec1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:21.811672 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08abe406-48e9-4dad-b81b-7b99d8d38ec1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:21.815688 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08abe406-48e9-4dad-b81b-7b99d8d38ec1 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:21.837308 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1155/2278] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:42:21 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:42:21.843434 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317f1d31-c9bf-4710-98c6-4e27f9597895 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:21.871156 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-317f1d31-c9bf-4710-98c6-4e27f9597895 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:21.889557 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32911522-6ad5-457a-a9b2-1b4d52b8d2da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:21.890621 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32911522-6ad5-457a-a9b2-1b4d52b8d2da admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:21.890786 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32911522-6ad5-457a-a9b2-1b4d52b8d2da admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:21.891197 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32911522-6ad5-457a-a9b2-1b4d52b8d2da admin admin] PATH_INFO: `/v3/projects/6ff840cf35e040419705695b892ac2f6/users/c959b1d755fe41c989d28e04497ecd6d/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:21.926849 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32911522-6ad5-457a-a9b2-1b4d52b8d2da admin admin] RBAC: Authorizing `identity:create_grant(project_id=6ff840cf35e040419705695b892ac2f6, user_id=c959b1d755fe41c989d28e04497ecd6d, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:21.955446 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32911522-6ad5-457a-a9b2-1b4d52b8d2da admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:22.101468 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1124/2279] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:42:21 2022] PUT /identity/v3/projects/6ff840cf35e040419705695b892ac2f6/users/c959b1d755fe41c989d28e04497ecd6d/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 259 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:42:22.113594 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91aeb2bb-0f11-41b5-a3ab-e362c61dd4db None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:22.113898 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91aeb2bb-0f11-41b5-a3ab-e362c61dd4db None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:22.114234 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91aeb2bb-0f11-41b5-a3ab-e362c61dd4db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:22.175033 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-91aeb2bb-0f11-41b5-a3ab-e362c61dd4db None None] MFA Rules not processed for user `c959b1d755fe41c989d28e04497ecd6d`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:42:22.176843 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-91aeb2bb-0f11-41b5-a3ab-e362c61dd4db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:22.380308 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1156/2280] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:42:22 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 268 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:42:22.398879 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32911522-6ad5-457a-a9b2-1b4d52b8d2da admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:22.401061 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-32911522-6ad5-457a-a9b2-1b4d52b8d2da admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:22.514789 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32911522-6ad5-457a-a9b2-1b4d52b8d2da admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:22.521657 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c586856-7f5f-4c55-a5af-96f46741af0f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:22.522660 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c586856-7f5f-4c55-a5af-96f46741af0f service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:22.523123 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c586856-7f5f-4c55-a5af-96f46741af0f service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:22.523811 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c586856-7f5f-4c55-a5af-96f46741af0f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:22.539183 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c586856-7f5f-4c55-a5af-96f46741af0f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:22.541150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c586856-7f5f-4c55-a5af-96f46741af0f service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:22.572467 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1125/2281] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:42:22 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:42:23.474519 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9ebc84-6805-46a8-b4e0-a774ea08225e None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:23.475404 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9ebc84-6805-46a8-b4e0-a774ea08225e None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:23.475404 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9ebc84-6805-46a8-b4e0-a774ea08225e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:23.494933 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-bc9ebc84-6805-46a8-b4e0-a774ea08225e None None] MFA Rules not processed for user `7b076f0335184c22a2aa07bf484c631e`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:42:23.496182 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-bc9ebc84-6805-46a8-b4e0-a774ea08225e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:23.513473 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:42:23.592332 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1157/2282] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:42:23 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:42:24.682665 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a41c3d-f750-497a-8e80-300a1786198c None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:24.683478 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a41c3d-f750-497a-8e80-300a1786198c None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:24.683641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a41c3d-f750-497a-8e80-300a1786198c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:24.732317 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-31a41c3d-f750-497a-8e80-300a1786198c None None] MFA Rules not processed for user `255cc0e9d9f748a19f8e104843259117`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:42:24.733955 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-31a41c3d-f750-497a-8e80-300a1786198c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:24.847855 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1126/2283] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:42:24 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:42:24.864803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9ebc84-6805-46a8-b4e0-a774ea08225e None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:24.867476 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-bc9ebc84-6805-46a8-b4e0-a774ea08225e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:24.948662 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9ebc84-6805-46a8-b4e0-a774ea08225e None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:24.956042 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0b3e8c-653d-4f8c-b6cb-fe5122314488 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4018d94a1feb4d6dade88ee9663f9729', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:24.956501 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0b3e8c-653d-4f8c-b6cb-fe5122314488 service glance] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:24.956905 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0b3e8c-653d-4f8c-b6cb-fe5122314488 service glance] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:24.957028 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0b3e8c-653d-4f8c-b6cb-fe5122314488 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:24.959124 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-6f0b3e8c-653d-4f8c-b6cb-fe5122314488 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:25.032221 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f0b3e8c-653d-4f8c-b6cb-fe5122314488 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:25.033870 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f0b3e8c-653d-4f8c-b6cb-fe5122314488 service glance] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:25.060695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1158/2284] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:42:24 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 197 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:42:25.286050 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1149a10-8233-4539-9a78-8506e785db71 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:25.287118 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1149a10-8233-4539-9a78-8506e785db71 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:25.288074 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1149a10-8233-4539-9a78-8506e785db71 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:25.291558 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1127/2285] 10.222.0.16 () {54 vars in 921 bytes} [Tue Jul 12 17:42:25 2022] GET /identity => generated 270 bytes in 15 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:42:25.299988 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc54eb88-6d37-4e59-a83c-ab32ba8d3db7 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:25.300294 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc54eb88-6d37-4e59-a83c-ab32ba8d3db7 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:25.300593 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc54eb88-6d37-4e59-a83c-ab32ba8d3db7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:25.371298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-fc54eb88-6d37-4e59-a83c-ab32ba8d3db7 None None] MFA Rules not processed for user `9012d5b88f554dc596582a7693c2a72a`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:42:25.376659 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-fc54eb88-6d37-4e59-a83c-ab32ba8d3db7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:25.499157 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aced7a4-fd64-4c60-9de8-1c1e5512edcd None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:25.499157 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aced7a4-fd64-4c60-9de8-1c1e5512edcd None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:25.499157 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aced7a4-fd64-4c60-9de8-1c1e5512edcd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:25.541710 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-4aced7a4-fd64-4c60-9de8-1c1e5512edcd None None] MFA Rules not processed for user `3ac0d5d8de4943a3888abee3b24b8a6f`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:42:25.543122 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-4aced7a4-fd64-4c60-9de8-1c1e5512edcd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:25.590633 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:42:25.661021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1159/2286] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Jul 12 17:42:25 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 362 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:42:25.803155 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc54eb88-6d37-4e59-a83c-ab32ba8d3db7 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:25.803155 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc54eb88-6d37-4e59-a83c-ab32ba8d3db7 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:25.803682 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1128/2287] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Jul 12 17:42:25 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:42:25.804057 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b81c01-394e-4850-b9d2-9d46394fdb71 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4018d94a1feb4d6dade88ee9663f9729', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:25.804057 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b81c01-394e-4850-b9d2-9d46394fdb71 service glance] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:25.804057 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b81c01-394e-4850-b9d2-9d46394fdb71 service glance] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:25.804057 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b81c01-394e-4850-b9d2-9d46394fdb71 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:25.804420 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aced7a4-fd64-4c60-9de8-1c1e5512edcd None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:25.804420 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-4aced7a4-fd64-4c60-9de8-1c1e5512edcd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:25.804589 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b81c01-394e-4850-b9d2-9d46394fdb71 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:25.804589 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b81c01-394e-4850-b9d2-9d46394fdb71 service glance] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:25.804589 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1160/2288] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:42:25 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 77 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:42:25.834220 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aced7a4-fd64-4c60-9de8-1c1e5512edcd None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:25.841577 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e071f47-5ea4-4956-90cf-d8693221e230 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7906d1a19c642bd93507a27bf3e3dae', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:25.842354 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e071f47-5ea4-4956-90cf-d8693221e230 service swift] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:25.842616 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e071f47-5ea4-4956-90cf-d8693221e230 service swift] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:25.842913 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e071f47-5ea4-4956-90cf-d8693221e230 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:25.853543 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e071f47-5ea4-4956-90cf-d8693221e230 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:25.856982 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e071f47-5ea4-4956-90cf-d8693221e230 service swift] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:25.871119 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1129/2289] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Jul 12 17:42:25 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 12 17:42:26.049907 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0bbba0-c5b5-4400-9673-6aa304792a9e None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:26.050537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0bbba0-c5b5-4400-9673-6aa304792a9e None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:26.050744 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0bbba0-c5b5-4400-9673-6aa304792a9e None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:26.052678 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1161/2290] 10.222.0.16 () {54 vars in 921 bytes} [Tue Jul 12 17:42:26 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:42:26.061856 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a7801b-97ce-4143-b985-9154af99a597 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:26.062706 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a7801b-97ce-4143-b985-9154af99a597 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:26.064929 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a7801b-97ce-4143-b985-9154af99a597 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:26.143833 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-93a7801b-97ce-4143-b985-9154af99a597 None None] MFA Rules not processed for user `bf5bf02ae13d4d86a1b157369ef07a00`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:42:26.147926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-93a7801b-97ce-4143-b985-9154af99a597 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:26.205454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=100 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:42:26.457196 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1130/2291] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Jul 12 17:42:26 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 398 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:42:26.499174 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0bbba0-c5b5-4400-9673-6aa304792a9e None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:26.502279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-ae0bbba0-c5b5-4400-9673-6aa304792a9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:26.530971 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0bbba0-c5b5-4400-9673-6aa304792a9e None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:26.539636 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-369599e6-3f7b-4a59-9ee8-48ca1ecd1dab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:26.540590 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369599e6-3f7b-4a59-9ee8-48ca1ecd1dab service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:26.541113 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369599e6-3f7b-4a59-9ee8-48ca1ecd1dab service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:26.542610 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-369599e6-3f7b-4a59-9ee8-48ca1ecd1dab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:26.563868 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-369599e6-3f7b-4a59-9ee8-48ca1ecd1dab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:26.565617 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-369599e6-3f7b-4a59-9ee8-48ca1ecd1dab service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:26.600258 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1162/2292] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:42:26 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:42:27.068933 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb24b2c-cf17-4631-8b9d-0bf8da124d42 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:27.069736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb24b2c-cf17-4631-8b9d-0bf8da124d42 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:27.069736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb24b2c-cf17-4631-8b9d-0bf8da124d42 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:27.071059 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1131/2293] 10.222.0.16 () {54 vars in 921 bytes} [Tue Jul 12 17:42:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:42:27.080747 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa47b4f-5d73-45fa-92dd-beef7aab5e00 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:27.081587 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa47b4f-5d73-45fa-92dd-beef7aab5e00 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:27.081587 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa47b4f-5d73-45fa-92dd-beef7aab5e00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:27.135729 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-6aa47b4f-5d73-45fa-92dd-beef7aab5e00 None None] MFA Rules not processed for user `9012d5b88f554dc596582a7693c2a72a`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:42:27.137656 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-6aa47b4f-5d73-45fa-92dd-beef7aab5e00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:27.193485 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1163/2294] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Jul 12 17:42:27 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:42:27.207497 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb24b2c-cf17-4631-8b9d-0bf8da124d42 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:27.209853 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-3fb24b2c-cf17-4631-8b9d-0bf8da124d42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:27.229907 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb24b2c-cf17-4631-8b9d-0bf8da124d42 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:27.237101 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e4eb69-b8c4-4dd4-bc83-488fb4eaeccb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:27.238767 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e4eb69-b8c4-4dd4-bc83-488fb4eaeccb service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:27.239580 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e4eb69-b8c4-4dd4-bc83-488fb4eaeccb service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:27.240237 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e4eb69-b8c4-4dd4-bc83-488fb4eaeccb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:27.263363 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16e4eb69-b8c4-4dd4-bc83-488fb4eaeccb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:27.273529 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16e4eb69-b8c4-4dd4-bc83-488fb4eaeccb service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:27.303323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1132/2295] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:42:27 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:42:37.237151 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=10 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:42:37.994467 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=78 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:42:40.392698 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9384c4-55e1-490d-9446-e290b1f288f9 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:40.393332 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9384c4-55e1-490d-9446-e290b1f288f9 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:40.393332 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9384c4-55e1-490d-9446-e290b1f288f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:40.412707 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-1f9384c4-55e1-490d-9446-e290b1f288f9 None None] MFA Rules not processed for user `ed4726967dd844b692fa7badd510267a`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:42:40.413867 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-1f9384c4-55e1-490d-9446-e290b1f288f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:40.421582 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1164/2296] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:42:40 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:42:43.552189 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08b4a4a-28bf-4a84-ab85-db0318cd4639 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:43.552783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08b4a4a-28bf-4a84-ab85-db0318cd4639 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:43.554266 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08b4a4a-28bf-4a84-ab85-db0318cd4639 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:43.603764 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-c08b4a4a-28bf-4a84-ab85-db0318cd4639 None None] MFA Rules not processed for user `3ac0d5d8de4943a3888abee3b24b8a6f`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:42:43.608630 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-c08b4a4a-28bf-4a84-ab85-db0318cd4639 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:43.643179 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1133/2297] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Jul 12 17:42:43 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:42:43.661219 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9384c4-55e1-490d-9446-e290b1f288f9 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:43.687083 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9384c4-55e1-490d-9446-e290b1f288f9 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:43.694522 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96aba897-b723-432e-871b-fea366c429b0 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7906d1a19c642bd93507a27bf3e3dae', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:43.695540 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96aba897-b723-432e-871b-fea366c429b0 service swift] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:43.696046 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96aba897-b723-432e-871b-fea366c429b0 service swift] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:43.696503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96aba897-b723-432e-871b-fea366c429b0 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:43.729389 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96aba897-b723-432e-871b-fea366c429b0 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:43.732765 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96aba897-b723-432e-871b-fea366c429b0 service swift] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:43.758264 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1165/2298] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Jul 12 17:42:43 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 12 17:42:53.594028 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:42:53.742775 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=9 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:42:57.076397 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8b8845-4a58-41e1-991e-ebbff85b53b3 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:57.076920 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8b8845-4a58-41e1-991e-ebbff85b53b3 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:57.076920 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8b8845-4a58-41e1-991e-ebbff85b53b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:57.101863 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-9c8b8845-4a58-41e1-991e-ebbff85b53b3 None None] MFA Rules not processed for user `3ac0d5d8de4943a3888abee3b24b8a6f`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:42:57.103151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-9c8b8845-4a58-41e1-991e-ebbff85b53b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:57.111397 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1134/2299] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Jul 12 17:42:57 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:42:57.127550 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96aba897-b723-432e-871b-fea366c429b0 service swift] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:57.152675 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96aba897-b723-432e-871b-fea366c429b0 service swift] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:57.160145 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a9585d-247a-4081-a20f-74a83e71e943 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7906d1a19c642bd93507a27bf3e3dae', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:57.161370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a9585d-247a-4081-a20f-74a83e71e943 service swift] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:57.161876 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a9585d-247a-4081-a20f-74a83e71e943 service swift] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:57.162370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a9585d-247a-4081-a20f-74a83e71e943 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:57.172665 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a9585d-247a-4081-a20f-74a83e71e943 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:57.176415 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a9585d-247a-4081-a20f-74a83e71e943 service swift] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:57.188279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1166/2300] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Jul 12 17:42:57 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 62 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 12 17:42:59.351065 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8b8845-4a58-41e1-991e-ebbff85b53b3 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:42:59.368995 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8b8845-4a58-41e1-991e-ebbff85b53b3 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:42:59.378784 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3703f7c6-bc11-4aaa-bb60-b9e11c8b12d4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:42:59.381623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3703f7c6-bc11-4aaa-bb60-b9e11c8b12d4 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:42:59.382094 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3703f7c6-bc11-4aaa-bb60-b9e11c8b12d4 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:42:59.382822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3703f7c6-bc11-4aaa-bb60-b9e11c8b12d4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:42:59.396689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-3703f7c6-bc11-4aaa-bb60-b9e11c8b12d4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:42:59.543405 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3703f7c6-bc11-4aaa-bb60-b9e11c8b12d4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:42:59.549800 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3703f7c6-bc11-4aaa-bb60-b9e11c8b12d4 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:42:59.565231 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1135/2301] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:42:59 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 215 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:43:07.183277 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=6 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:43:09.557781 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=33 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:43:09.721034 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a9585d-247a-4081-a20f-74a83e71e943 service swift] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:43:09.799412 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a9585d-247a-4081-a20f-74a83e71e943 service swift] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:43:09.804622 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-263e2b5e-f31f-4e1f-be06-6cb3105f0926 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:43:09.805423 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263e2b5e-f31f-4e1f-be06-6cb3105f0926 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:43:09.805749 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263e2b5e-f31f-4e1f-be06-6cb3105f0926 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:43:09.806057 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-263e2b5e-f31f-4e1f-be06-6cb3105f0926 admin admin] PATH_INFO: `/v3/users/c959b1d755fe41c989d28e04497ecd6d` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:43:09.807398 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-263e2b5e-f31f-4e1f-be06-6cb3105f0926 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c959b1d755fe41c989d28e04497ecd6d)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:43:09.809479 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-263e2b5e-f31f-4e1f-be06-6cb3105f0926 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:43:10.030124 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-263e2b5e-f31f-4e1f-be06-6cb3105f0926 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c959b1d755fe41c989d28e04497ecd6d'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:43:10.030670 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-263e2b5e-f31f-4e1f-be06-6cb3105f0926 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c959b1d755fe41c989d28e04497ecd6d'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:43:10.031802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-263e2b5e-f31f-4e1f-be06-6cb3105f0926 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c959b1d755fe41c989d28e04497ecd6d'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:43:10.064723 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-263e2b5e-f31f-4e1f-be06-6cb3105f0926 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c959b1d755fe41c989d28e04497ecd6d'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:43:10.089353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-263e2b5e-f31f-4e1f-be06-6cb3105f0926 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c959b1d755fe41c989d28e04497ecd6d'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:43:10.185730 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1167/2302] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:43:09 2022] DELETE /identity/v3/users/c959b1d755fe41c989d28e04497ecd6d => generated 0 bytes in 465 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:43:10.190460 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407485f2-7b2a-474e-8c67-6929563b3045 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:43:10.190859 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407485f2-7b2a-474e-8c67-6929563b3045 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:43:10.191992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407485f2-7b2a-474e-8c67-6929563b3045 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:43:10.223684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-407485f2-7b2a-474e-8c67-6929563b3045 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:43:10.225240 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-407485f2-7b2a-474e-8c67-6929563b3045 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:43:10.299568 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1136/2303] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:43:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:43:10.310019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-263e2b5e-f31f-4e1f-be06-6cb3105f0926 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:43:10.311518 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-263e2b5e-f31f-4e1f-be06-6cb3105f0926 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:43:10.373708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-263e2b5e-f31f-4e1f-be06-6cb3105f0926 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:43:10.377496 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6dcab9-b3e4-4e7b-b47d-a188745407f1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:43:10.378150 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6dcab9-b3e4-4e7b-b47d-a188745407f1 service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:43:10.378454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6dcab9-b3e4-4e7b-b47d-a188745407f1 service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:43:10.379259 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6dcab9-b3e4-4e7b-b47d-a188745407f1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:43:10.388899 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d6dcab9-b3e4-4e7b-b47d-a188745407f1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:43:10.390344 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d6dcab9-b3e4-4e7b-b47d-a188745407f1 service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:43:10.401561 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1168/2304] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:43:10 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:43:11.013524 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407485f2-7b2a-474e-8c67-6929563b3045 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:43:11.015849 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-407485f2-7b2a-474e-8c67-6929563b3045 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:43:11.031434 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407485f2-7b2a-474e-8c67-6929563b3045 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:43:11.037533 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55de3dec-b89a-45cd-b7f1-a2078d473c67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:43:11.038628 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55de3dec-b89a-45cd-b7f1-a2078d473c67 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:43:11.038917 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55de3dec-b89a-45cd-b7f1-a2078d473c67 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:43:11.039227 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55de3dec-b89a-45cd-b7f1-a2078d473c67 admin admin] PATH_INFO: `/v3/projects/6ff840cf35e040419705695b892ac2f6` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:43:11.040711 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55de3dec-b89a-45cd-b7f1-a2078d473c67 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6ff840cf35e040419705695b892ac2f6)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:43:11.042719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55de3dec-b89a-45cd-b7f1-a2078d473c67 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:43:11.143704 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-55de3dec-b89a-45cd-b7f1-a2078d473c67 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6ff840cf35e040419705695b892ac2f6', 'request_id': 'req-55de3dec-b89a-45cd-b7f1-a2078d473c67'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:43:11.150313 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-55de3dec-b89a-45cd-b7f1-a2078d473c67 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6ff840cf35e040419705695b892ac2f6', 'request_id': 'req-55de3dec-b89a-45cd-b7f1-a2078d473c67'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:43:11.156000 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-55de3dec-b89a-45cd-b7f1-a2078d473c67 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6ff840cf35e040419705695b892ac2f6', 'request_id': 'req-55de3dec-b89a-45cd-b7f1-a2078d473c67'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:43:11.223206 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-55de3dec-b89a-45cd-b7f1-a2078d473c67 admin admin] The token cache is being invalidate because project 6ff840cf35e040419705695b892ac2f6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:43:11.223505 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-55de3dec-b89a-45cd-b7f1-a2078d473c67 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:43:11.224161 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-55de3dec-b89a-45cd-b7f1-a2078d473c67 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:43:11.225757 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1137/2305] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:43:11 2022] DELETE /identity/v3/projects/6ff840cf35e040419705695b892ac2f6 => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:43:11.230943 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fdc71b-8e81-4b92-9caa-ffe8162dbcf3 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:43:11.231416 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fdc71b-8e81-4b92-9caa-ffe8162dbcf3 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:43:11.231549 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1fdc71b-8e81-4b92-9caa-ffe8162dbcf3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:43:11.250320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-a1fdc71b-8e81-4b92-9caa-ffe8162dbcf3 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:43:11.251575 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-a1fdc71b-8e81-4b92-9caa-ffe8162dbcf3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:43:11.270048 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:43:11.316477 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1169/2306] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:43:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:43:11.325335 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0611368a-0bb4-4d9d-9c74-79f18fa865dc None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:43:11.325546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0611368a-0bb4-4d9d-9c74-79f18fa865dc None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:43:11.325723 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0611368a-0bb4-4d9d-9c74-79f18fa865dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:43:11.347073 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-0611368a-0bb4-4d9d-9c74-79f18fa865dc None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:43:11.349314 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-0611368a-0bb4-4d9d-9c74-79f18fa865dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:43:11.358521 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1138/2307] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:43:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:43:11.362785 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1fdc71b-8e81-4b92-9caa-ffe8162dbcf3 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:43:11.373188 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1fdc71b-8e81-4b92-9caa-ffe8162dbcf3 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:43:11.378048 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b611e4c-b59e-49dd-8f8a-4f0adc41ff74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:43:11.378794 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b611e4c-b59e-49dd-8f8a-4f0adc41ff74 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:43:11.379085 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b611e4c-b59e-49dd-8f8a-4f0adc41ff74 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:43:11.379367 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b611e4c-b59e-49dd-8f8a-4f0adc41ff74 admin admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:43:11.380106 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b611e4c-b59e-49dd-8f8a-4f0adc41ff74 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:43:11.380441 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b611e4c-b59e-49dd-8f8a-4f0adc41ff74 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:43:11.382162 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b611e4c-b59e-49dd-8f8a-4f0adc41ff74 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:43:11.396146 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1170/2308] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:43:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:43:11.400391 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0611368a-0bb4-4d9d-9c74-79f18fa865dc None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:43:11.409794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0611368a-0bb4-4d9d-9c74-79f18fa865dc None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:43:11.415163 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d38dcdc-eb45-4289-9d07-fd74e15e971a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:43:11.415830 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d38dcdc-eb45-4289-9d07-fd74e15e971a admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:43:11.416114 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d38dcdc-eb45-4289-9d07-fd74e15e971a admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:43:11.416392 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d38dcdc-eb45-4289-9d07-fd74e15e971a admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:43:11.417090 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d38dcdc-eb45-4289-9d07-fd74e15e971a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:43:11.419014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d38dcdc-eb45-4289-9d07-fd74e15e971a admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:43:11.510598 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1139/2309] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:43:11 2022] POST /identity/v3/projects => generated 371 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:43:11.514640 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b611e4c-b59e-49dd-8f8a-4f0adc41ff74 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:43:11.526545 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b611e4c-b59e-49dd-8f8a-4f0adc41ff74 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:43:11.594639 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e25423-fbe5-4b93-9602-3a1895c2e1fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:43:11.595314 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e25423-fbe5-4b93-9602-3a1895c2e1fc admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:43:11.595599 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e25423-fbe5-4b93-9602-3a1895c2e1fc admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:43:11.595873 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e25423-fbe5-4b93-9602-3a1895c2e1fc admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:43:11.596561 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e25423-fbe5-4b93-9602-3a1895c2e1fc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:43:11.599307 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e25423-fbe5-4b93-9602-3a1895c2e1fc admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:43:11.710402 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1171/2310] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:43:11 2022] POST /identity/v3/users => generated 286 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:43:11.714581 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d38dcdc-eb45-4289-9d07-fd74e15e971a admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:43:11.726177 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d38dcdc-eb45-4289-9d07-fd74e15e971a admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:43:11.731142 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d94eb7a-5ee4-4f1f-8776-85af39034b2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:43:11.731825 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d94eb7a-5ee4-4f1f-8776-85af39034b2c admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:43:11.732116 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d94eb7a-5ee4-4f1f-8776-85af39034b2c admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:43:11.732410 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d94eb7a-5ee4-4f1f-8776-85af39034b2c admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:43:11.733150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d94eb7a-5ee4-4f1f-8776-85af39034b2c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:43:11.734786 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d94eb7a-5ee4-4f1f-8776-85af39034b2c admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:43:11.746118 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1140/2311] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:43:11 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:43:11.750209 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e25423-fbe5-4b93-9602-3a1895c2e1fc admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:43:11.761503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e25423-fbe5-4b93-9602-3a1895c2e1fc admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:43:11.766191 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6891e6c5-6a51-4e8a-88e0-5282a9b5f1e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:43:11.766873 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6891e6c5-6a51-4e8a-88e0-5282a9b5f1e2 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:43:11.767168 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6891e6c5-6a51-4e8a-88e0-5282a9b5f1e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:43:11.767453 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6891e6c5-6a51-4e8a-88e0-5282a9b5f1e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:43:11.768198 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6891e6c5-6a51-4e8a-88e0-5282a9b5f1e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:43:11.770657 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6891e6c5-6a51-4e8a-88e0-5282a9b5f1e2 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:43:11.781388 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1172/2312] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:43:11 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:43:11.785475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d94eb7a-5ee4-4f1f-8776-85af39034b2c admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:43:11.798555 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d94eb7a-5ee4-4f1f-8776-85af39034b2c admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:43:11.803762 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e2e737-9769-457f-a688-6f38478ab8c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:43:11.804673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e2e737-9769-457f-a688-6f38478ab8c0 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:43:11.804959 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e2e737-9769-457f-a688-6f38478ab8c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:43:11.805240 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e2e737-9769-457f-a688-6f38478ab8c0 admin admin] PATH_INFO: `/v3/projects/0967218fbd6e41fca773c6e82dec133e/users/89008b07b9fd457d8bf24350f40b6143/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:43:11.824132 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e2e737-9769-457f-a688-6f38478ab8c0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0967218fbd6e41fca773c6e82dec133e, user_id=89008b07b9fd457d8bf24350f40b6143, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:43:11.826829 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e2e737-9769-457f-a688-6f38478ab8c0 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:43:11.916621 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1141/2313] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:43:11 2022] PUT /identity/v3/projects/0967218fbd6e41fca773c6e82dec133e/users/89008b07b9fd457d8bf24350f40b6143/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:43:11.931262 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520dc867-0ac7-4671-a849-1ab459cca892 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:43:11.931824 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520dc867-0ac7-4671-a849-1ab459cca892 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:43:11.932285 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520dc867-0ac7-4671-a849-1ab459cca892 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:43:11.967079 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-520dc867-0ac7-4671-a849-1ab459cca892 None None] MFA Rules not processed for user `89008b07b9fd457d8bf24350f40b6143`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:43:11.968437 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-520dc867-0ac7-4671-a849-1ab459cca892 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:43:12.062492 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1173/2314] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:43:11 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:43:12.071821 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e2e737-9769-457f-a688-6f38478ab8c0 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:43:12.073791 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-f4e2e737-9769-457f-a688-6f38478ab8c0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:43:12.105144 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:43:12.165233 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e2e737-9769-457f-a688-6f38478ab8c0 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:43:12.169085 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f87a1d-74a4-498e-a9ad-eea1a7eed1bc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:43:12.169830 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f87a1d-74a4-498e-a9ad-eea1a7eed1bc service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:43:12.170124 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f87a1d-74a4-498e-a9ad-eea1a7eed1bc service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:43:12.170460 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f87a1d-74a4-498e-a9ad-eea1a7eed1bc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:43:12.181512 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f87a1d-74a4-498e-a9ad-eea1a7eed1bc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:43:12.183723 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f87a1d-74a4-498e-a9ad-eea1a7eed1bc service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:43:12.195995 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1142/2315] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:43:12 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:43:12.232114 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520dc867-0ac7-4671-a849-1ab459cca892 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:43:12.256041 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:43:12.303083 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520dc867-0ac7-4671-a849-1ab459cca892 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:43:12.307711 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e258237b-726d-4939-b24b-5a659d061641 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:43:12.308579 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e258237b-726d-4939-b24b-5a659d061641 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:43:12.308920 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e258237b-726d-4939-b24b-5a659d061641 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:43:12.309185 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e258237b-726d-4939-b24b-5a659d061641 admin admin] PATH_INFO: `/v3/users/89008b07b9fd457d8bf24350f40b6143` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:43:12.310395 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e258237b-726d-4939-b24b-5a659d061641 admin admin] RBAC: Authorizing `identity:delete_user(user_id=89008b07b9fd457d8bf24350f40b6143)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:43:12.312357 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e258237b-726d-4939-b24b-5a659d061641 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:43:12.535271 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e258237b-726d-4939-b24b-5a659d061641 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '89008b07b9fd457d8bf24350f40b6143'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:43:12.536040 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e258237b-726d-4939-b24b-5a659d061641 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '89008b07b9fd457d8bf24350f40b6143'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:43:12.536662 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e258237b-726d-4939-b24b-5a659d061641 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '89008b07b9fd457d8bf24350f40b6143'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:43:12.543868 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e258237b-726d-4939-b24b-5a659d061641 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '89008b07b9fd457d8bf24350f40b6143'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:43:12.557946 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e258237b-726d-4939-b24b-5a659d061641 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '89008b07b9fd457d8bf24350f40b6143'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:43:12.681772 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1174/2316] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:43:12 2022] DELETE /identity/v3/users/89008b07b9fd457d8bf24350f40b6143 => generated 0 bytes in 450 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:43:12.686550 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469e59ae-f02b-4404-99d1-b17ebe892da5 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:43:12.686864 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469e59ae-f02b-4404-99d1-b17ebe892da5 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:43:12.687133 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469e59ae-f02b-4404-99d1-b17ebe892da5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:43:12.704824 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-469e59ae-f02b-4404-99d1-b17ebe892da5 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:43:12.706035 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-469e59ae-f02b-4404-99d1-b17ebe892da5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:43:12.768698 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1143/2317] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:43:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:43:12.779051 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e258237b-726d-4939-b24b-5a659d061641 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:43:12.780778 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-e258237b-726d-4939-b24b-5a659d061641 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:43:12.841870 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e258237b-726d-4939-b24b-5a659d061641 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:43:12.845630 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a22141e-15df-4404-b475-ce44982812f0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:43:12.846325 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a22141e-15df-4404-b475-ce44982812f0 service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:43:12.846640 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a22141e-15df-4404-b475-ce44982812f0 service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:43:12.846928 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a22141e-15df-4404-b475-ce44982812f0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:43:12.857058 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a22141e-15df-4404-b475-ce44982812f0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:43:12.858572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a22141e-15df-4404-b475-ce44982812f0 service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:43:12.869769 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1175/2318] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:43:12 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:43:13.425032 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469e59ae-f02b-4404-99d1-b17ebe892da5 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:43:13.427052 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-469e59ae-f02b-4404-99d1-b17ebe892da5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:43:13.440427 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469e59ae-f02b-4404-99d1-b17ebe892da5 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:43:13.448378 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cdf088d-320c-4773-84bd-681681d55d99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:43:13.449614 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cdf088d-320c-4773-84bd-681681d55d99 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:43:13.449905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cdf088d-320c-4773-84bd-681681d55d99 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:43:13.450192 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cdf088d-320c-4773-84bd-681681d55d99 admin admin] PATH_INFO: `/v3/projects/0967218fbd6e41fca773c6e82dec133e` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:43:13.451479 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cdf088d-320c-4773-84bd-681681d55d99 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0967218fbd6e41fca773c6e82dec133e)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:43:13.453600 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cdf088d-320c-4773-84bd-681681d55d99 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:43:13.611708 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-5cdf088d-320c-4773-84bd-681681d55d99 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0967218fbd6e41fca773c6e82dec133e', 'request_id': 'req-5cdf088d-320c-4773-84bd-681681d55d99'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:43:13.629643 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-5cdf088d-320c-4773-84bd-681681d55d99 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0967218fbd6e41fca773c6e82dec133e', 'request_id': 'req-5cdf088d-320c-4773-84bd-681681d55d99'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:43:13.639221 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-5cdf088d-320c-4773-84bd-681681d55d99 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0967218fbd6e41fca773c6e82dec133e', 'request_id': 'req-5cdf088d-320c-4773-84bd-681681d55d99'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:43:13.722107 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-5cdf088d-320c-4773-84bd-681681d55d99 admin admin] The token cache is being invalidate because project 0967218fbd6e41fca773c6e82dec133e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:43:13.722535 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-5cdf088d-320c-4773-84bd-681681d55d99 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:43:13.723321 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-5cdf088d-320c-4773-84bd-681681d55d99 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:43:13.726289 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1144/2319] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:43:13 2022] DELETE /identity/v3/projects/0967218fbd6e41fca773c6e82dec133e => generated 0 bytes in 302 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:43:18.095814 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80606869-99ce-4b88-b47b-7e583b9770d1 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:43:18.096485 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80606869-99ce-4b88-b47b-7e583b9770d1 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:43:18.098099 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80606869-99ce-4b88-b47b-7e583b9770d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:43:18.118196 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-80606869-99ce-4b88-b47b-7e583b9770d1 None None] MFA Rules not processed for user `3ac0d5d8de4943a3888abee3b24b8a6f`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:43:18.119737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-80606869-99ce-4b88-b47b-7e583b9770d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:43:18.159392 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:43:18.183531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1176/2320] 10.222.0.16 () {58 vars in 967 bytes} [Tue Jul 12 17:43:18 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:43:18.199157 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cdf088d-320c-4773-84bd-681681d55d99 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:43:18.200650 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-5cdf088d-320c-4773-84bd-681681d55d99 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:43:18.267584 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:43:18.291163 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cdf088d-320c-4773-84bd-681681d55d99 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:43:18.296925 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e47f8a6-213a-45a7-8a39-5d91e7a713b7 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7906d1a19c642bd93507a27bf3e3dae', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:43:18.297695 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e47f8a6-213a-45a7-8a39-5d91e7a713b7 service swift] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:43:18.297978 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e47f8a6-213a-45a7-8a39-5d91e7a713b7 service swift] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:43:18.298354 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e47f8a6-213a-45a7-8a39-5d91e7a713b7 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:43:18.312120 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e47f8a6-213a-45a7-8a39-5d91e7a713b7 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:43:18.312120 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e47f8a6-213a-45a7-8a39-5d91e7a713b7 service swift] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:43:18.319174 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1145/2321] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Jul 12 17:43:18 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 12 17:43:22.339261 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee76a0d4-6eeb-4966-b175-8b522dc43f95 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:43:22.340069 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee76a0d4-6eeb-4966-b175-8b522dc43f95 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:43:22.340364 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee76a0d4-6eeb-4966-b175-8b522dc43f95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:43:22.362168 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-ee76a0d4-6eeb-4966-b175-8b522dc43f95 None None] MFA Rules not processed for user `3ac0d5d8de4943a3888abee3b24b8a6f`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:43:22.364251 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-ee76a0d4-6eeb-4966-b175-8b522dc43f95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:43:22.373446 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1177/2322] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Jul 12 17:43:22 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:43:22.387699 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e47f8a6-213a-45a7-8a39-5d91e7a713b7 service swift] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:43:22.400199 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e47f8a6-213a-45a7-8a39-5d91e7a713b7 service swift] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:43:22.406162 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7265ba08-1831-4726-8f6e-61d9f551dbe3 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7906d1a19c642bd93507a27bf3e3dae', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:43:22.407200 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7265ba08-1831-4726-8f6e-61d9f551dbe3 service swift] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:43:22.407685 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7265ba08-1831-4726-8f6e-61d9f551dbe3 service swift] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:43:22.408179 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7265ba08-1831-4726-8f6e-61d9f551dbe3 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:43:22.419611 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7265ba08-1831-4726-8f6e-61d9f551dbe3 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:43:22.421434 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7265ba08-1831-4726-8f6e-61d9f551dbe3 service swift] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:43:22.433780 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1146/2323] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Jul 12 17:43:22 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 12 17:43:23.386158 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344c3d36-8b7a-4918-bf0c-bab54884b4a6 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:43:23.386948 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344c3d36-8b7a-4918-bf0c-bab54884b4a6 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:43:23.387106 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344c3d36-8b7a-4918-bf0c-bab54884b4a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:43:23.410618 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-344c3d36-8b7a-4918-bf0c-bab54884b4a6 None None] MFA Rules not processed for user `c8441a4a72154ee3bbb95af66abf2ffd`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:43:23.411973 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-344c3d36-8b7a-4918-bf0c-bab54884b4a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:43:23.494842 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1178/2324] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:43:23 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:43:23.839132 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394ab406-3c75-4039-886d-9603df74ea01 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:43:23.839666 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394ab406-3c75-4039-886d-9603df74ea01 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:43:23.839747 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394ab406-3c75-4039-886d-9603df74ea01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:43:23.859890 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-394ab406-3c75-4039-886d-9603df74ea01 None None] MFA Rules not processed for user `3ac0d5d8de4943a3888abee3b24b8a6f`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:43:23.861152 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-394ab406-3c75-4039-886d-9603df74ea01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:43:23.868872 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1147/2325] 10.222.0.16 () {58 vars in 967 bytes} [Tue Jul 12 17:43:23 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:43:23.882135 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344c3d36-8b7a-4918-bf0c-bab54884b4a6 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:43:23.897609 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344c3d36-8b7a-4918-bf0c-bab54884b4a6 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:43:23.903553 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12dd357-00b7-4700-80fa-f393531ef402 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7906d1a19c642bd93507a27bf3e3dae', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:43:23.904234 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12dd357-00b7-4700-80fa-f393531ef402 service swift] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:43:23.904526 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12dd357-00b7-4700-80fa-f393531ef402 service swift] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:43:23.904805 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12dd357-00b7-4700-80fa-f393531ef402 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:43:23.914770 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12dd357-00b7-4700-80fa-f393531ef402 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:43:23.916382 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12dd357-00b7-4700-80fa-f393531ef402 service swift] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:43:23.927334 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1179/2326] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Jul 12 17:43:23 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 12 17:43:24.928315 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a6aa64-c5f5-47af-a622-b35b260fda83 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:43:24.928315 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a6aa64-c5f5-47af-a622-b35b260fda83 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:43:24.928315 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a6aa64-c5f5-47af-a622-b35b260fda83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:43:24.949412 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-56a6aa64-c5f5-47af-a622-b35b260fda83 None None] MFA Rules not processed for user `3ac0d5d8de4943a3888abee3b24b8a6f`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:43:24.951758 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-56a6aa64-c5f5-47af-a622-b35b260fda83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:43:24.961628 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1148/2327] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Jul 12 17:43:24 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:43:24.980011 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12dd357-00b7-4700-80fa-f393531ef402 service swift] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:43:24.996165 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12dd357-00b7-4700-80fa-f393531ef402 service swift] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:43:25.003801 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b149689b-f6fb-4712-a6a9-52d1fceb3c45 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7906d1a19c642bd93507a27bf3e3dae', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:43:25.005587 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b149689b-f6fb-4712-a6a9-52d1fceb3c45 service swift] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:43:25.006056 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b149689b-f6fb-4712-a6a9-52d1fceb3c45 service swift] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:43:25.006392 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b149689b-f6fb-4712-a6a9-52d1fceb3c45 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:43:25.023732 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b149689b-f6fb-4712-a6a9-52d1fceb3c45 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:43:25.025770 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b149689b-f6fb-4712-a6a9-52d1fceb3c45 service swift] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:43:25.037121 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1180/2328] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Jul 12 17:43:24 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 58 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 12 17:43:25.777794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483a0a3b-f072-4686-99d5-eae3d8a551fa None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:43:25.778078 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483a0a3b-f072-4686-99d5-eae3d8a551fa None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:43:25.778676 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483a0a3b-f072-4686-99d5-eae3d8a551fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:43:25.814385 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-483a0a3b-f072-4686-99d5-eae3d8a551fa None None] MFA Rules not processed for user `3ac0d5d8de4943a3888abee3b24b8a6f`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:43:25.817671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-483a0a3b-f072-4686-99d5-eae3d8a551fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:43:25.840369 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1149/2329] 10.222.0.16 () {58 vars in 967 bytes} [Tue Jul 12 17:43:25 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 65 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:43:25.855616 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b149689b-f6fb-4712-a6a9-52d1fceb3c45 service swift] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:43:25.868397 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b149689b-f6fb-4712-a6a9-52d1fceb3c45 service swift] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:43:25.876151 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54464932-d80d-4357-a4bf-282aa73fd958 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7906d1a19c642bd93507a27bf3e3dae', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:43:25.877095 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54464932-d80d-4357-a4bf-282aa73fd958 service swift] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:43:25.877392 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54464932-d80d-4357-a4bf-282aa73fd958 service swift] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:43:25.877855 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54464932-d80d-4357-a4bf-282aa73fd958 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:43:25.890289 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54464932-d80d-4357-a4bf-282aa73fd958 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:43:25.893284 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54464932-d80d-4357-a4bf-282aa73fd958 service swift] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:43:25.902578 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1181/2330] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Jul 12 17:43:25 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 12 17:43:28.937028 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483a0a3b-f072-4686-99d5-eae3d8a551fa None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:43:28.938648 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-483a0a3b-f072-4686-99d5-eae3d8a551fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:43:29.020560 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483a0a3b-f072-4686-99d5-eae3d8a551fa None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:43:29.026705 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d997fc-eca7-4dfd-baa5-b99ff1987abc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:43:29.027721 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d997fc-eca7-4dfd-baa5-b99ff1987abc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:43:29.028112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d997fc-eca7-4dfd-baa5-b99ff1987abc admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:43:29.028401 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d997fc-eca7-4dfd-baa5-b99ff1987abc admin admin] PATH_INFO: `/v3/users/7b076f0335184c22a2aa07bf484c631e` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:43:29.029956 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d997fc-eca7-4dfd-baa5-b99ff1987abc admin admin] RBAC: Authorizing `identity:delete_user(user_id=7b076f0335184c22a2aa07bf484c631e)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:43:29.032567 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d997fc-eca7-4dfd-baa5-b99ff1987abc admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:43:29.215235 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-78d997fc-eca7-4dfd-baa5-b99ff1987abc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7b076f0335184c22a2aa07bf484c631e'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:43:29.315403 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-78d997fc-eca7-4dfd-baa5-b99ff1987abc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7b076f0335184c22a2aa07bf484c631e'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:43:29.316207 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-78d997fc-eca7-4dfd-baa5-b99ff1987abc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7b076f0335184c22a2aa07bf484c631e'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:43:29.327997 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-78d997fc-eca7-4dfd-baa5-b99ff1987abc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7b076f0335184c22a2aa07bf484c631e'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:43:29.348536 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-78d997fc-eca7-4dfd-baa5-b99ff1987abc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7b076f0335184c22a2aa07bf484c631e'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:43:29.358819 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1150/2331] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:43:28 2022] DELETE /identity/v3/users/7b076f0335184c22a2aa07bf484c631e => generated 0 bytes in 422 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:43:29.364320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ba2122-adf4-4474-aa4f-5a2bd0dcf129 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:43:29.364915 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ba2122-adf4-4474-aa4f-5a2bd0dcf129 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:43:29.365210 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ba2122-adf4-4474-aa4f-5a2bd0dcf129 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:43:29.387148 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-d5ba2122-adf4-4474-aa4f-5a2bd0dcf129 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:43:29.399706 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-d5ba2122-adf4-4474-aa4f-5a2bd0dcf129 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:43:29.531599 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1182/2332] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:43:29 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:43:29.551287 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d997fc-eca7-4dfd-baa5-b99ff1987abc admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:43:29.554123 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-78d997fc-eca7-4dfd-baa5-b99ff1987abc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:43:29.598580 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:43:29.655652 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d997fc-eca7-4dfd-baa5-b99ff1987abc admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:43:29.664782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63dcaedd-7109-4783-8b85-54aec4ac5da1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:43:29.666056 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63dcaedd-7109-4783-8b85-54aec4ac5da1 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:43:29.666722 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63dcaedd-7109-4783-8b85-54aec4ac5da1 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:43:29.667611 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63dcaedd-7109-4783-8b85-54aec4ac5da1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:43:29.685389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63dcaedd-7109-4783-8b85-54aec4ac5da1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:43:29.688160 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63dcaedd-7109-4783-8b85-54aec4ac5da1 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:43:29.706421 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1151/2333] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:43:29 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:43:30.081107 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5ba2122-adf4-4474-aa4f-5a2bd0dcf129 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:43:30.083612 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-d5ba2122-adf4-4474-aa4f-5a2bd0dcf129 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:43:30.109487 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5ba2122-adf4-4474-aa4f-5a2bd0dcf129 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:43:30.119294 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba78ad5-3738-4538-babb-93732fc91531 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:43:30.120755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba78ad5-3738-4538-babb-93732fc91531 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:43:30.121272 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba78ad5-3738-4538-babb-93732fc91531 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:43:30.121773 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba78ad5-3738-4538-babb-93732fc91531 admin admin] PATH_INFO: `/v3/projects/019d9f28339c4316bb3cda3e4c6ea908` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:43:30.124219 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba78ad5-3738-4538-babb-93732fc91531 admin admin] RBAC: Authorizing `identity:delete_project(project_id=019d9f28339c4316bb3cda3e4c6ea908)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:43:30.127699 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba78ad5-3738-4538-babb-93732fc91531 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:43:30.248573 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:43:30.264074 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-aba78ad5-3738-4538-babb-93732fc91531 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '019d9f28339c4316bb3cda3e4c6ea908', 'request_id': 'req-aba78ad5-3738-4538-babb-93732fc91531'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:43:30.276594 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-aba78ad5-3738-4538-babb-93732fc91531 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '019d9f28339c4316bb3cda3e4c6ea908', 'request_id': 'req-aba78ad5-3738-4538-babb-93732fc91531'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:43:30.338395 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-aba78ad5-3738-4538-babb-93732fc91531 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '019d9f28339c4316bb3cda3e4c6ea908', 'request_id': 'req-aba78ad5-3738-4538-babb-93732fc91531'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:43:30.364967 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-aba78ad5-3738-4538-babb-93732fc91531 admin admin] The token cache is being invalidate because project 019d9f28339c4316bb3cda3e4c6ea908 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:43:30.365586 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-aba78ad5-3738-4538-babb-93732fc91531 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:43:30.366665 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-aba78ad5-3738-4538-babb-93732fc91531 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:43:30.369666 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1183/2334] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:43:30 2022] DELETE /identity/v3/projects/019d9f28339c4316bb3cda3e4c6ea908 => generated 0 bytes in 289 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:43:34.989838 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63dcaedd-7109-4783-8b85-54aec4ac5da1 service neutron] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:43:34.999792 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63dcaedd-7109-4783-8b85-54aec4ac5da1 service neutron] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:43:35.066763 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de6fb7d-cae8-42af-b75c-d64c173f0fab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:43:35.067500 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de6fb7d-cae8-42af-b75c-d64c173f0fab service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:43:35.067793 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de6fb7d-cae8-42af-b75c-d64c173f0fab service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:43:35.068080 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de6fb7d-cae8-42af-b75c-d64c173f0fab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:43:35.070496 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-6de6fb7d-cae8-42af-b75c-d64c173f0fab service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:43:35.167820 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6de6fb7d-cae8-42af-b75c-d64c173f0fab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:43:35.169498 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6de6fb7d-cae8-42af-b75c-d64c173f0fab service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:43:35.182840 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1152/2335] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:43:34 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 194 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:43:40.356149 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=10,keystone:DELETE=4,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:43:43.564042 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a5a0ec-eade-41c6-98ef-eeaf7bd0c7b6 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:43:43.565177 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a5a0ec-eade-41c6-98ef-eeaf7bd0c7b6 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:43:43.565640 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a5a0ec-eade-41c6-98ef-eeaf7bd0c7b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:43:43.591349 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-d4a5a0ec-eade-41c6-98ef-eeaf7bd0c7b6 None None] MFA Rules not processed for user `ce09df98c929416e8ca67b0728349bfb`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:43:43.592623 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-d4a5a0ec-eade-41c6-98ef-eeaf7bd0c7b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:43:43.675269 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1184/2336] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:43:43 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:43:45.175741 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=66 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:43:53.664787 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=25 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:44:03.797829 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de6fb7d-cae8-42af-b75c-d64c173f0fab service neutron] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:44:03.799667 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-6de6fb7d-cae8-42af-b75c-d64c173f0fab service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:44:03.871330 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de6fb7d-cae8-42af-b75c-d64c173f0fab service neutron] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:44:03.874974 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a184089f-8bf7-4787-ac65-fe643a8c4cd9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:44:03.875644 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a184089f-8bf7-4787-ac65-fe643a8c4cd9 service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:44:03.875943 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a184089f-8bf7-4787-ac65-fe643a8c4cd9 service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:44:03.876341 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a184089f-8bf7-4787-ac65-fe643a8c4cd9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:44:03.878090 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-a184089f-8bf7-4787-ac65-fe643a8c4cd9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:44:03.955389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a184089f-8bf7-4787-ac65-fe643a8c4cd9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:44:03.956956 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a184089f-8bf7-4787-ac65-fe643a8c4cd9 service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:44:03.970077 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1153/2337] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:44:03 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 173 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:44:07.485541 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a5a0ec-eade-41c6-98ef-eeaf7bd0c7b6 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:44:07.487536 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-d4a5a0ec-eade-41c6-98ef-eeaf7bd0c7b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:44:07.501050 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a5a0ec-eade-41c6-98ef-eeaf7bd0c7b6 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:44:07.505528 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2fe6a9-81f8-41ad-9f3d-0233321487ca service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:44:07.506242 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2fe6a9-81f8-41ad-9f3d-0233321487ca service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:44:07.506562 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2fe6a9-81f8-41ad-9f3d-0233321487ca service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:44:07.506848 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2fe6a9-81f8-41ad-9f3d-0233321487ca service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:44:07.508714 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-9a2fe6a9-81f8-41ad-9f3d-0233321487ca service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:44:07.523796 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2fe6a9-81f8-41ad-9f3d-0233321487ca service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:44:07.525253 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2fe6a9-81f8-41ad-9f3d-0233321487ca service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:44:07.537207 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1185/2338] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:44:07 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:44:13.964621 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=54 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:44:14.878209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754fcab2-cf42-43e2-8df1-7475e6b780bf None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:44:14.878609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754fcab2-cf42-43e2-8df1-7475e6b780bf None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:44:14.879006 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754fcab2-cf42-43e2-8df1-7475e6b780bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:44:14.902182 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-754fcab2-cf42-43e2-8df1-7475e6b780bf None None] MFA Rules not processed for user `9feeaf257bb441b7a14b57f1f493d08f`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:44:14.902182 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-754fcab2-cf42-43e2-8df1-7475e6b780bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:44:14.960898 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1154/2339] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:44:14 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:44:17.531689 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=6 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:44:18.967420 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2fe6a9-81f8-41ad-9f3d-0233321487ca service neutron] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:44:18.969739 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-9a2fe6a9-81f8-41ad-9f3d-0233321487ca service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:44:18.984811 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2fe6a9-81f8-41ad-9f3d-0233321487ca service neutron] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:44:18.997543 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712386fe-9341-41a2-b17f-f645fbf5210f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:44:18.998407 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712386fe-9341-41a2-b17f-f645fbf5210f service nova] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:44:18.998738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712386fe-9341-41a2-b17f-f645fbf5210f service nova] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:44:18.999046 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712386fe-9341-41a2-b17f-f645fbf5210f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:44:19.000917 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-712386fe-9341-41a2-b17f-f645fbf5210f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:44:19.013516 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712386fe-9341-41a2-b17f-f645fbf5210f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:44:19.015125 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712386fe-9341-41a2-b17f-f645fbf5210f service nova] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:44:19.029130 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1186/2340] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:44:18 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 62 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:44:22.494609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754fcab2-cf42-43e2-8df1-7475e6b780bf None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:44:22.496499 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-754fcab2-cf42-43e2-8df1-7475e6b780bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:44:22.525535 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754fcab2-cf42-43e2-8df1-7475e6b780bf None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:44:22.551150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c48b5ba-962f-4486-8473-b1c8db68531f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:44:22.551150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c48b5ba-962f-4486-8473-b1c8db68531f service nova] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:44:22.551150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c48b5ba-962f-4486-8473-b1c8db68531f service nova] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:44:22.551150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c48b5ba-962f-4486-8473-b1c8db68531f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:44:22.552207 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-7c48b5ba-962f-4486-8473-b1c8db68531f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:44:22.608451 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c48b5ba-962f-4486-8473-b1c8db68531f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:44:22.610561 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c48b5ba-962f-4486-8473-b1c8db68531f service nova] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:44:22.621838 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1155/2341] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:44:22 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:44:28.092779 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712386fe-9341-41a2-b17f-f645fbf5210f service nova] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:44:28.095202 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-712386fe-9341-41a2-b17f-f645fbf5210f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:44:28.109020 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712386fe-9341-41a2-b17f-f645fbf5210f service nova] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:44:28.113665 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e4460a2-29ce-4956-ba1e-14974243cce0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:44:28.114146 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4460a2-29ce-4956-ba1e-14974243cce0 service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:44:28.114754 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4460a2-29ce-4956-ba1e-14974243cce0 service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:44:28.115637 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e4460a2-29ce-4956-ba1e-14974243cce0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:44:28.117808 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-4e4460a2-29ce-4956-ba1e-14974243cce0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:44:28.131318 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e4460a2-29ce-4956-ba1e-14974243cce0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:44:28.132955 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e4460a2-29ce-4956-ba1e-14974243cce0 service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:44:28.144545 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1187/2342] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:44:28 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:44:32.617007 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=53 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:44:38.139126 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=12 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:44:40.219520 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c48b5ba-962f-4486-8473-b1c8db68531f service nova] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:44:40.231770 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c48b5ba-962f-4486-8473-b1c8db68531f service nova] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:44:40.237786 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ee4676c-c0b5-4eee-9ea4-6ebfb576f4a2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:44:40.238519 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee4676c-c0b5-4eee-9ea4-6ebfb576f4a2 service nova] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:44:40.238815 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee4676c-c0b5-4eee-9ea4-6ebfb576f4a2 service nova] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:44:40.239105 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee4676c-c0b5-4eee-9ea4-6ebfb576f4a2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:44:40.243284 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-3ee4676c-c0b5-4eee-9ea4-6ebfb576f4a2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:44:40.258993 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ee4676c-c0b5-4eee-9ea4-6ebfb576f4a2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:44:40.260926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ee4676c-c0b5-4eee-9ea4-6ebfb576f4a2 service nova] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:44:40.282646 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1156/2343] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:44:40 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:44:47.363505 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8794dd-c540-4ffa-897b-a065f9f28383 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:44:47.364080 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8794dd-c540-4ffa-897b-a065f9f28383 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:44:47.364080 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8794dd-c540-4ffa-897b-a065f9f28383 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:44:47.383467 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-fa8794dd-c540-4ffa-897b-a065f9f28383 None None] MFA Rules not processed for user `c8441a4a72154ee3bbb95af66abf2ffd`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:44:47.384690 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-fa8794dd-c540-4ffa-897b-a065f9f28383 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:44:47.391976 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1188/2344] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:44:47 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:44:47.721680 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ee4676c-c0b5-4eee-9ea4-6ebfb576f4a2 service nova] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:44:47.723573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-3ee4676c-c0b5-4eee-9ea4-6ebfb576f4a2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:44:47.737618 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ee4676c-c0b5-4eee-9ea4-6ebfb576f4a2 service nova] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:44:47.742732 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453cbbbc-ec71-43b7-87bc-691e5069cd97 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:44:47.743592 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453cbbbc-ec71-43b7-87bc-691e5069cd97 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:44:47.743908 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453cbbbc-ec71-43b7-87bc-691e5069cd97 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:44:47.744190 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453cbbbc-ec71-43b7-87bc-691e5069cd97 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:44:47.745996 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-453cbbbc-ec71-43b7-87bc-691e5069cd97 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:44:47.756608 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453cbbbc-ec71-43b7-87bc-691e5069cd97 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:44:47.758033 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453cbbbc-ec71-43b7-87bc-691e5069cd97 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:44:47.766392 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1157/2345] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:44:47 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:44:48.202080 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8794dd-c540-4ffa-897b-a065f9f28383 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:44:48.203633 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-fa8794dd-c540-4ffa-897b-a065f9f28383 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:44:48.289131 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8794dd-c540-4ffa-897b-a065f9f28383 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:44:48.294225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873306e7-b737-4316-9cc2-05dafa5d3f1e service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f623be28c83247c78d3ea6873c347fe0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:44:48.294917 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873306e7-b737-4316-9cc2-05dafa5d3f1e service placement] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:44:48.295207 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873306e7-b737-4316-9cc2-05dafa5d3f1e service placement] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:44:48.295490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873306e7-b737-4316-9cc2-05dafa5d3f1e service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:44:48.297420 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-873306e7-b737-4316-9cc2-05dafa5d3f1e service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:44:48.312255 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-873306e7-b737-4316-9cc2-05dafa5d3f1e service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:44:48.313863 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-873306e7-b737-4316-9cc2-05dafa5d3f1e service placement] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:44:48.326581 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1189/2346] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:44:48 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:44:55.096533 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453cbbbc-ec71-43b7-87bc-691e5069cd97 service neutron] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:44:55.106782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453cbbbc-ec71-43b7-87bc-691e5069cd97 service neutron] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:44:55.110611 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a732a7-c856-418b-b826-03efd4f2b877 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:44:55.111303 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a732a7-c856-418b-b826-03efd4f2b877 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:44:55.111597 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a732a7-c856-418b-b826-03efd4f2b877 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:44:55.111882 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a732a7-c856-418b-b826-03efd4f2b877 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:44:55.113953 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-f1a732a7-c856-418b-b826-03efd4f2b877 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:44:55.125659 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a732a7-c856-418b-b826-03efd4f2b877 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:44:55.127384 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a732a7-c856-418b-b826-03efd4f2b877 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:44:55.138174 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1158/2347] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:44:55 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:44:56.172157 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873306e7-b737-4316-9cc2-05dafa5d3f1e service placement] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:44:56.187311 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873306e7-b737-4316-9cc2-05dafa5d3f1e service placement] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:44:56.196053 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a380d95-1d30-4955-b447-0bbbf0d7c7b6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:44:56.197169 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a380d95-1d30-4955-b447-0bbbf0d7c7b6 service nova] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:44:56.198223 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a380d95-1d30-4955-b447-0bbbf0d7c7b6 service nova] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:44:56.198787 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a380d95-1d30-4955-b447-0bbbf0d7c7b6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:44:56.204196 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-4a380d95-1d30-4955-b447-0bbbf0d7c7b6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:44:56.218864 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a380d95-1d30-4955-b447-0bbbf0d7c7b6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:44:56.220837 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a380d95-1d30-4955-b447-0bbbf0d7c7b6 service nova] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:44:56.236413 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1190/2348] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:44:56 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:44:57.528941 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a732a7-c856-418b-b826-03efd4f2b877 service neutron] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:44:57.537805 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a732a7-c856-418b-b826-03efd4f2b877 service neutron] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:44:57.541636 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc1d33ef-3e77-4ede-84fb-7f8060e9be4f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:44:57.542292 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc1d33ef-3e77-4ede-84fb-7f8060e9be4f service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:44:57.542616 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc1d33ef-3e77-4ede-84fb-7f8060e9be4f service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:44:57.542902 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc1d33ef-3e77-4ede-84fb-7f8060e9be4f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:44:57.544693 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-cc1d33ef-3e77-4ede-84fb-7f8060e9be4f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:44:57.619993 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc1d33ef-3e77-4ede-84fb-7f8060e9be4f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:44:57.621374 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc1d33ef-3e77-4ede-84fb-7f8060e9be4f service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:44:57.633354 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1159/2349] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:44:57 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:45:01.556365 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a380d95-1d30-4955-b447-0bbbf0d7c7b6 service nova] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:45:01.573710 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a380d95-1d30-4955-b447-0bbbf0d7c7b6 service nova] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:45:01.579559 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab23040d-61e2-4cd1-aebc-1b9ff92bb1d4 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f623be28c83247c78d3ea6873c347fe0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:45:01.581639 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab23040d-61e2-4cd1-aebc-1b9ff92bb1d4 service placement] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:01.582178 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab23040d-61e2-4cd1-aebc-1b9ff92bb1d4 service placement] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:01.582724 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab23040d-61e2-4cd1-aebc-1b9ff92bb1d4 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:01.585675 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-ab23040d-61e2-4cd1-aebc-1b9ff92bb1d4 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:45:01.607428 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab23040d-61e2-4cd1-aebc-1b9ff92bb1d4 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:45:01.609086 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab23040d-61e2-4cd1-aebc-1b9ff92bb1d4 service placement] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:45:01.621415 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1191/2350] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:45:01 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 66 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:45:01.986726 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc1d33ef-3e77-4ede-84fb-7f8060e9be4f service neutron] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:45:01.988270 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-cc1d33ef-3e77-4ede-84fb-7f8060e9be4f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:45:02.000948 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc1d33ef-3e77-4ede-84fb-7f8060e9be4f service neutron] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:45:02.005702 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab1ef20-6277-4248-a3a3-262bb36f65f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:45:02.006714 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab1ef20-6277-4248-a3a3-262bb36f65f3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:02.006999 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab1ef20-6277-4248-a3a3-262bb36f65f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:02.007325 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab1ef20-6277-4248-a3a3-262bb36f65f3 admin admin] PATH_INFO: `/v3/users/ce09df98c929416e8ca67b0728349bfb` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:02.008547 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab1ef20-6277-4248-a3a3-262bb36f65f3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ce09df98c929416e8ca67b0728349bfb)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:45:02.010517 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab1ef20-6277-4248-a3a3-262bb36f65f3 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:45:02.256700 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-0ab1ef20-6277-4248-a3a3-262bb36f65f3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ce09df98c929416e8ca67b0728349bfb'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:45:02.327275 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-0ab1ef20-6277-4248-a3a3-262bb36f65f3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ce09df98c929416e8ca67b0728349bfb'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:45:02.327902 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-0ab1ef20-6277-4248-a3a3-262bb36f65f3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ce09df98c929416e8ca67b0728349bfb'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:45:02.336754 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-0ab1ef20-6277-4248-a3a3-262bb36f65f3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ce09df98c929416e8ca67b0728349bfb'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:45:02.348938 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-0ab1ef20-6277-4248-a3a3-262bb36f65f3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ce09df98c929416e8ca67b0728349bfb'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:45:02.355245 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1160/2351] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:45:01 2022] DELETE /identity/v3/users/ce09df98c929416e8ca67b0728349bfb => generated 0 bytes in 369 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:45:02.603907 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab23040d-61e2-4cd1-aebc-1b9ff92bb1d4 service placement] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:45:02.617178 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab23040d-61e2-4cd1-aebc-1b9ff92bb1d4 service placement] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:45:02.685485 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9e9ca1-1d25-4782-8cbe-685b991c1784 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:45:02.686509 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9e9ca1-1d25-4782-8cbe-685b991c1784 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:02.686771 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9e9ca1-1d25-4782-8cbe-685b991c1784 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:02.687053 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9e9ca1-1d25-4782-8cbe-685b991c1784 admin admin] PATH_INFO: `/v3/projects/497c73f9a78743e9b24b2b6ff47d7ede` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:02.688269 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee9e9ca1-1d25-4782-8cbe-685b991c1784 admin admin] RBAC: Authorizing `identity:delete_project(project_id=497c73f9a78743e9b24b2b6ff47d7ede)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:45:02.690358 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee9e9ca1-1d25-4782-8cbe-685b991c1784 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:45:02.855750 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ee9e9ca1-1d25-4782-8cbe-685b991c1784 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '497c73f9a78743e9b24b2b6ff47d7ede', 'request_id': 'req-ee9e9ca1-1d25-4782-8cbe-685b991c1784'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:45:02.861549 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ee9e9ca1-1d25-4782-8cbe-685b991c1784 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '497c73f9a78743e9b24b2b6ff47d7ede', 'request_id': 'req-ee9e9ca1-1d25-4782-8cbe-685b991c1784'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:45:02.865414 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:45:02.866613 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab1ef20-6277-4248-a3a3-262bb36f65f3 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:45:02.868578 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-0ab1ef20-6277-4248-a3a3-262bb36f65f3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:45:02.874233 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=78,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:45:02.943485 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab1ef20-6277-4248-a3a3-262bb36f65f3 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:45:02.956028 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ee9e9ca1-1d25-4782-8cbe-685b991c1784 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '497c73f9a78743e9b24b2b6ff47d7ede', 'request_id': 'req-ee9e9ca1-1d25-4782-8cbe-685b991c1784'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:45:02.971753 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ee9e9ca1-1d25-4782-8cbe-685b991c1784 admin admin] The token cache is being invalidate because project 497c73f9a78743e9b24b2b6ff47d7ede was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:45:02.972041 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ee9e9ca1-1d25-4782-8cbe-685b991c1784 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:45:02.972713 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-ee9e9ca1-1d25-4782-8cbe-685b991c1784 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:45:02.979181 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1192/2352] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:45:02 2022] DELETE /identity/v3/projects/497c73f9a78743e9b24b2b6ff47d7ede => generated 0 bytes in 371 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:45:02.980304 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f17719-84b8-4440-8f6f-ae2836d43128 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:02.980602 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f17719-84b8-4440-8f6f-ae2836d43128 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:02.980938 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f17719-84b8-4440-8f6f-ae2836d43128 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:03.000740 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-77f17719-84b8-4440-8f6f-ae2836d43128 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:45:03.002056 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-77f17719-84b8-4440-8f6f-ae2836d43128 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:45:03.016119 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49ff719-c202-401e-a290-4bfe504da458 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:45:03.016951 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49ff719-c202-401e-a290-4bfe504da458 service nova] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:03.017244 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49ff719-c202-401e-a290-4bfe504da458 service nova] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:03.017524 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49ff719-c202-401e-a290-4bfe504da458 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:03.019743 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-a49ff719-c202-401e-a290-4bfe504da458 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:45:03.034416 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a49ff719-c202-401e-a290-4bfe504da458 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:45:03.036741 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a49ff719-c202-401e-a290-4bfe504da458 service nova] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:45:03.050941 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1161/2353] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:45:02 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 186 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:45:03.072666 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1193/2354] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:45:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:45:03.081710 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a90d9d1-d38e-4157-bbb8-07cd9a9fc7fb None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:03.082094 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a90d9d1-d38e-4157-bbb8-07cd9a9fc7fb None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:03.082094 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a90d9d1-d38e-4157-bbb8-07cd9a9fc7fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:03.104378 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-2a90d9d1-d38e-4157-bbb8-07cd9a9fc7fb None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:45:03.105621 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-2a90d9d1-d38e-4157-bbb8-07cd9a9fc7fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:45:03.115007 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1162/2355] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:45:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:45:03.119665 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f17719-84b8-4440-8f6f-ae2836d43128 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:45:03.129774 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f17719-84b8-4440-8f6f-ae2836d43128 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:45:03.135645 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94a382a-c262-4da4-8434-8a614747e818 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:45:03.136517 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94a382a-c262-4da4-8434-8a614747e818 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:03.139991 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94a382a-c262-4da4-8434-8a614747e818 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:03.139991 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94a382a-c262-4da4-8434-8a614747e818 admin admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:03.139991 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94a382a-c262-4da4-8434-8a614747e818 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:45:03.139991 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94a382a-c262-4da4-8434-8a614747e818 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:45:03.140513 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94a382a-c262-4da4-8434-8a614747e818 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:45:03.156762 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1194/2356] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:45:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:45:03.161191 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a90d9d1-d38e-4157-bbb8-07cd9a9fc7fb None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:45:03.172152 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a90d9d1-d38e-4157-bbb8-07cd9a9fc7fb None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:45:03.176943 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef547f1f-ec43-4463-8693-f7205099cb8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:45:03.177634 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef547f1f-ec43-4463-8693-f7205099cb8f admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:03.177936 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef547f1f-ec43-4463-8693-f7205099cb8f admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:03.178804 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef547f1f-ec43-4463-8693-f7205099cb8f admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:03.179519 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef547f1f-ec43-4463-8693-f7205099cb8f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:45:03.181431 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef547f1f-ec43-4463-8693-f7205099cb8f admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:45:03.317822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1163/2357] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:45:03 2022] POST /identity/v3/projects => generated 377 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:45:03.323263 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94a382a-c262-4da4-8434-8a614747e818 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:45:03.334137 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94a382a-c262-4da4-8434-8a614747e818 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:45:03.411536 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a4dfa2-18c4-4f87-8ac1-f544fb42a579 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:45:03.412517 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a4dfa2-18c4-4f87-8ac1-f544fb42a579 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:03.412993 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a4dfa2-18c4-4f87-8ac1-f544fb42a579 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:03.413386 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a4dfa2-18c4-4f87-8ac1-f544fb42a579 admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:03.414461 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a4dfa2-18c4-4f87-8ac1-f544fb42a579 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:45:03.417221 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a4dfa2-18c4-4f87-8ac1-f544fb42a579 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:45:03.549269 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1195/2358] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:45:03 2022] POST /identity/v3/users => generated 289 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:45:03.550494 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef547f1f-ec43-4463-8693-f7205099cb8f admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:45:03.562493 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef547f1f-ec43-4463-8693-f7205099cb8f admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:45:03.567383 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798b0bcb-04ee-4d12-a3f5-3fac6c8da059 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:45:03.568083 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798b0bcb-04ee-4d12-a3f5-3fac6c8da059 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:03.568371 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798b0bcb-04ee-4d12-a3f5-3fac6c8da059 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:03.568650 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798b0bcb-04ee-4d12-a3f5-3fac6c8da059 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:03.569677 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798b0bcb-04ee-4d12-a3f5-3fac6c8da059 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:45:03.571368 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798b0bcb-04ee-4d12-a3f5-3fac6c8da059 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:45:03.582863 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1164/2359] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:45:03 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:45:03.587334 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a4dfa2-18c4-4f87-8ac1-f544fb42a579 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:45:03.598447 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a4dfa2-18c4-4f87-8ac1-f544fb42a579 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:45:03.603871 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8e6f9d-d9d5-4dc8-899d-021453b774a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:45:03.604557 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8e6f9d-d9d5-4dc8-899d-021453b774a2 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:03.605127 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8e6f9d-d9d5-4dc8-899d-021453b774a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:03.605127 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8e6f9d-d9d5-4dc8-899d-021453b774a2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:03.605844 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8e6f9d-d9d5-4dc8-899d-021453b774a2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:45:03.607470 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8e6f9d-d9d5-4dc8-899d-021453b774a2 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:45:03.618657 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1196/2360] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:45:03 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:45:03.623176 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798b0bcb-04ee-4d12-a3f5-3fac6c8da059 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:45:03.634838 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798b0bcb-04ee-4d12-a3f5-3fac6c8da059 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:45:03.639959 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23ffe8a-af30-4f14-afbc-b715da4cceb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:45:03.640931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23ffe8a-af30-4f14-afbc-b715da4cceb2 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:03.641223 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23ffe8a-af30-4f14-afbc-b715da4cceb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:03.641534 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23ffe8a-af30-4f14-afbc-b715da4cceb2 admin admin] PATH_INFO: `/v3/projects/041c2c26c82a44d98d2229969c1f3618/users/e3d86a1cde9f407cae8ac050fe000d47/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:03.657369 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23ffe8a-af30-4f14-afbc-b715da4cceb2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=041c2c26c82a44d98d2229969c1f3618, user_id=e3d86a1cde9f407cae8ac050fe000d47, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:45:03.660105 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23ffe8a-af30-4f14-afbc-b715da4cceb2 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:45:03.745737 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1165/2361] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:45:03 2022] PUT /identity/v3/projects/041c2c26c82a44d98d2229969c1f3618/users/e3d86a1cde9f407cae8ac050fe000d47/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:45:03.751085 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af50f161-33b7-4bce-b1c9-17d2c607f127 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:03.751385 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af50f161-33b7-4bce-b1c9-17d2c607f127 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:03.751701 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af50f161-33b7-4bce-b1c9-17d2c607f127 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:03.770388 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-af50f161-33b7-4bce-b1c9-17d2c607f127 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:45:03.771737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-af50f161-33b7-4bce-b1c9-17d2c607f127 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:45:03.827245 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:INSERT=4,keystone:DELETE=1,keystone:SELECT=95 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:45:03.840583 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1197/2362] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:45:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:45:03.856655 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23ffe8a-af30-4f14-afbc-b715da4cceb2 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:45:03.858638 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-a23ffe8a-af30-4f14-afbc-b715da4cceb2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:45:03.888161 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:45:03.920040 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23ffe8a-af30-4f14-afbc-b715da4cceb2 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:45:03.923867 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ba27c7-4afb-401b-af05-f48ddc6f3550 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:45:03.924530 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ba27c7-4afb-401b-af05-f48ddc6f3550 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:03.924823 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ba27c7-4afb-401b-af05-f48ddc6f3550 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:03.925109 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ba27c7-4afb-401b-af05-f48ddc6f3550 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:03.936360 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ba27c7-4afb-401b-af05-f48ddc6f3550 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:45:03.937867 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ba27c7-4afb-401b-af05-f48ddc6f3550 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:45:03.949581 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1166/2363] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:45:03 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:45:10.040710 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1523573f-271b-4c57-b6b9-dd1ea6908e17 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:10.041635 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1523573f-271b-4c57-b6b9-dd1ea6908e17 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:10.041921 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1523573f-271b-4c57-b6b9-dd1ea6908e17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:10.076506 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-1523573f-271b-4c57-b6b9-dd1ea6908e17 None None] MFA Rules not processed for user `e3d86a1cde9f407cae8ac050fe000d47`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:45:10.076506 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-1523573f-271b-4c57-b6b9-dd1ea6908e17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:45:10.158863 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1198/2364] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:45:10 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:45:10.173833 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ba27c7-4afb-401b-af05-f48ddc6f3550 service neutron] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:45:10.175463 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-85ba27c7-4afb-401b-af05-f48ddc6f3550 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:45:10.258393 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ba27c7-4afb-401b-af05-f48ddc6f3550 service neutron] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:45:10.264000 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1381993-7489-4971-8a57-c8f0f468b15a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:45:10.264662 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1381993-7489-4971-8a57-c8f0f468b15a service nova] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:10.265026 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1381993-7489-4971-8a57-c8f0f468b15a service nova] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:10.265322 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1381993-7489-4971-8a57-c8f0f468b15a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:10.275741 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1381993-7489-4971-8a57-c8f0f468b15a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:45:10.277521 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1381993-7489-4971-8a57-c8f0f468b15a service nova] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:45:10.288267 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1167/2365] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:45:10 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:45:14.530979 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c5dcbb-e490-4b87-a5b6-eff7585bd082 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:14.531953 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c5dcbb-e490-4b87-a5b6-eff7585bd082 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:14.533973 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c5dcbb-e490-4b87-a5b6-eff7585bd082 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:14.566832 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-c0c5dcbb-e490-4b87-a5b6-eff7585bd082 None None] MFA Rules not processed for user `e3d86a1cde9f407cae8ac050fe000d47`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:45:14.569291 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-c0c5dcbb-e490-4b87-a5b6-eff7585bd082 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:45:14.588020 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1199/2366] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:45:14 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:45:15.745772 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bd6324-c6c6-4ea2-a856-028cacd7e5c2 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:15.746346 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bd6324-c6c6-4ea2-a856-028cacd7e5c2 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:15.746346 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bd6324-c6c6-4ea2-a856-028cacd7e5c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:15.765124 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-54bd6324-c6c6-4ea2-a856-028cacd7e5c2 None None] MFA Rules not processed for user `bde6ee9df10146d98991b0d4bb37fc48`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:45:15.766452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-54bd6324-c6c6-4ea2-a856-028cacd7e5c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:45:15.836559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1168/2367] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:45:15 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:45:21.997641 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4d22b4-ec17-4d08-a7d2-987236ddc911 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:21.998719 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4d22b4-ec17-4d08-a7d2-987236ddc911 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:21.999208 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4d22b4-ec17-4d08-a7d2-987236ddc911 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:22.022295 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-7a4d22b4-ec17-4d08-a7d2-987236ddc911 None None] MFA Rules not processed for user `3ac0d5d8de4943a3888abee3b24b8a6f`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:45:22.023969 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-7a4d22b4-ec17-4d08-a7d2-987236ddc911 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:45:22.137125 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1200/2368] 10.222.0.16 () {58 vars in 967 bytes} [Tue Jul 12 17:45:21 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:45:22.288333 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54bd6324-c6c6-4ea2-a856-028cacd7e5c2 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:45:22.291891 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-54bd6324-c6c6-4ea2-a856-028cacd7e5c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:45:22.363928 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:45:22.383051 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54bd6324-c6c6-4ea2-a856-028cacd7e5c2 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:45:22.389992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6913fde9-da9c-4700-95a5-6925db377552 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7906d1a19c642bd93507a27bf3e3dae', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:45:22.390697 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6913fde9-da9c-4700-95a5-6925db377552 service swift] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:22.390985 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6913fde9-da9c-4700-95a5-6925db377552 service swift] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:22.391471 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6913fde9-da9c-4700-95a5-6925db377552 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:22.403929 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6913fde9-da9c-4700-95a5-6925db377552 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:45:22.406080 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6913fde9-da9c-4700-95a5-6925db377552 service swift] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:45:22.414692 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1169/2369] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Jul 12 17:45:22 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 12 17:45:30.638468 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c700b0e-3acd-4b08-8125-af9b1d1622fd None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:30.638949 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c700b0e-3acd-4b08-8125-af9b1d1622fd None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:30.639122 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c700b0e-3acd-4b08-8125-af9b1d1622fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:30.658233 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-3c700b0e-3acd-4b08-8125-af9b1d1622fd None None] MFA Rules not processed for user `3ac0d5d8de4943a3888abee3b24b8a6f`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:45:30.659516 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-3c700b0e-3acd-4b08-8125-af9b1d1622fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:45:30.667298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1201/2370] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Jul 12 17:45:30 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:45:30.680052 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6913fde9-da9c-4700-95a5-6925db377552 service swift] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:45:30.692703 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6913fde9-da9c-4700-95a5-6925db377552 service swift] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:45:30.698939 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a902dd5d-7f26-49da-9c82-2735fed8db20 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7906d1a19c642bd93507a27bf3e3dae', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:45:30.699607 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a902dd5d-7f26-49da-9c82-2735fed8db20 service swift] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:30.701441 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a902dd5d-7f26-49da-9c82-2735fed8db20 service swift] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:30.701441 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a902dd5d-7f26-49da-9c82-2735fed8db20 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:30.709324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a902dd5d-7f26-49da-9c82-2735fed8db20 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:45:30.710972 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a902dd5d-7f26-49da-9c82-2735fed8db20 service swift] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:45:30.720023 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1170/2371] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Jul 12 17:45:30 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 12 17:45:32.609478 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99414942-8dcf-4219-be95-b3afae3fba3d None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:32.610550 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99414942-8dcf-4219-be95-b3afae3fba3d None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:32.611043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99414942-8dcf-4219-be95-b3afae3fba3d None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:32.614795 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1202/2372] 10.222.0.16 () {54 vars in 921 bytes} [Tue Jul 12 17:45:32 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:45:32.629535 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f699e0f6-900e-452d-b35c-c870911022e8 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:32.630140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f699e0f6-900e-452d-b35c-c870911022e8 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:32.630140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f699e0f6-900e-452d-b35c-c870911022e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:32.663330 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-f699e0f6-900e-452d-b35c-c870911022e8 None None] MFA Rules not processed for user `f623be28c83247c78d3ea6873c347fe0`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:45:32.665495 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-f699e0f6-900e-452d-b35c-c870911022e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:45:32.779130 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1171/2373] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Jul 12 17:45:32 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:45:32.827823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99414942-8dcf-4219-be95-b3afae3fba3d None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:45:32.832058 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-99414942-8dcf-4219-be95-b3afae3fba3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:45:32.860738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f699e0f6-900e-452d-b35c-c870911022e8 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:45:32.862961 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-f699e0f6-900e-452d-b35c-c870911022e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:45:32.887020 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99414942-8dcf-4219-be95-b3afae3fba3d None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:45:32.897275 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4002498-b870-40b4-8131-21a71f9bec64 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f623be28c83247c78d3ea6873c347fe0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:45:32.898583 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f699e0f6-900e-452d-b35c-c870911022e8 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:45:32.899400 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4002498-b870-40b4-8131-21a71f9bec64 service placement] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:32.899788 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4002498-b870-40b4-8131-21a71f9bec64 service placement] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:32.900911 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4002498-b870-40b4-8131-21a71f9bec64 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:32.916732 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa9e046f-1b79-4bcf-829e-805c63f3f171 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:45:32.917776 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9e046f-1b79-4bcf-829e-805c63f3f171 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:32.922521 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9e046f-1b79-4bcf-829e-805c63f3f171 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:32.922521 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9e046f-1b79-4bcf-829e-805c63f3f171 admin admin] PATH_INFO: `/v3/users/ed4726967dd844b692fa7badd510267a` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:32.924437 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa9e046f-1b79-4bcf-829e-805c63f3f171 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ed4726967dd844b692fa7badd510267a)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:45:32.933958 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa9e046f-1b79-4bcf-829e-805c63f3f171 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:45:32.935466 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4002498-b870-40b4-8131-21a71f9bec64 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:45:32.941291 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4002498-b870-40b4-8131-21a71f9bec64 service placement] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:45:32.965737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1203/2374] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:45:32 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:45:33.226709 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-aa9e046f-1b79-4bcf-829e-805c63f3f171 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ed4726967dd844b692fa7badd510267a'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:45:33.413876 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-aa9e046f-1b79-4bcf-829e-805c63f3f171 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ed4726967dd844b692fa7badd510267a'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:45:33.414986 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-aa9e046f-1b79-4bcf-829e-805c63f3f171 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ed4726967dd844b692fa7badd510267a'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:45:33.427270 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-aa9e046f-1b79-4bcf-829e-805c63f3f171 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ed4726967dd844b692fa7badd510267a'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:45:33.447129 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-aa9e046f-1b79-4bcf-829e-805c63f3f171 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ed4726967dd844b692fa7badd510267a'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:45:33.456282 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1172/2375] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:45:32 2022] DELETE /identity/v3/users/ed4726967dd844b692fa7badd510267a => generated 0 bytes in 596 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:45:33.933362 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4002498-b870-40b4-8131-21a71f9bec64 service placement] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:45:33.939631 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=73 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:45:33.950096 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4002498-b870-40b4-8131-21a71f9bec64 service placement] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:45:34.088588 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1cd82d0-d867-47dd-806f-5f7b61b6d216 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:45:34.089955 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1cd82d0-d867-47dd-806f-5f7b61b6d216 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:34.090520 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1cd82d0-d867-47dd-806f-5f7b61b6d216 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:34.091057 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1cd82d0-d867-47dd-806f-5f7b61b6d216 admin admin] PATH_INFO: `/v3/projects/80285308991b47308701fd3363bec122` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:34.093601 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1cd82d0-d867-47dd-806f-5f7b61b6d216 admin admin] RBAC: Authorizing `identity:delete_project(project_id=80285308991b47308701fd3363bec122)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:45:34.095876 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1cd82d0-d867-47dd-806f-5f7b61b6d216 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:45:34.876424 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-b1cd82d0-d867-47dd-806f-5f7b61b6d216 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '80285308991b47308701fd3363bec122', 'request_id': 'req-b1cd82d0-d867-47dd-806f-5f7b61b6d216'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:45:34.884551 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-b1cd82d0-d867-47dd-806f-5f7b61b6d216 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '80285308991b47308701fd3363bec122', 'request_id': 'req-b1cd82d0-d867-47dd-806f-5f7b61b6d216'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:45:35.042209 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-b1cd82d0-d867-47dd-806f-5f7b61b6d216 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '80285308991b47308701fd3363bec122', 'request_id': 'req-b1cd82d0-d867-47dd-806f-5f7b61b6d216'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:45:35.057132 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-b1cd82d0-d867-47dd-806f-5f7b61b6d216 admin admin] The token cache is being invalidate because project 80285308991b47308701fd3363bec122 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:45:35.057539 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-b1cd82d0-d867-47dd-806f-5f7b61b6d216 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:45:35.058296 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-b1cd82d0-d867-47dd-806f-5f7b61b6d216 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:45:35.060027 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1204/2376] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:45:33 2022] DELETE /identity/v3/projects/80285308991b47308701fd3363bec122 => generated 0 bytes in 1127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:45:35.064432 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa9e046f-1b79-4bcf-829e-805c63f3f171 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:45:35.071958 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-aa9e046f-1b79-4bcf-829e-805c63f3f171 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:45:35.083114 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa9e046f-1b79-4bcf-829e-805c63f3f171 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:45:35.156369 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:45:35.190132 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5bbcd06-1a4d-4de2-8d20-556cb4da5ef4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:45:35.191181 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bbcd06-1a4d-4de2-8d20-556cb4da5ef4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:35.191554 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bbcd06-1a4d-4de2-8d20-556cb4da5ef4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:35.192015 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bbcd06-1a4d-4de2-8d20-556cb4da5ef4 admin admin] PATH_INFO: `/v3/users/255cc0e9d9f748a19f8e104843259117` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:35.195552 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5bbcd06-1a4d-4de2-8d20-556cb4da5ef4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=255cc0e9d9f748a19f8e104843259117)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:45:35.200301 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5bbcd06-1a4d-4de2-8d20-556cb4da5ef4 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:45:35.709552 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-d5bbcd06-1a4d-4de2-8d20-556cb4da5ef4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '255cc0e9d9f748a19f8e104843259117'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:45:35.830802 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-d5bbcd06-1a4d-4de2-8d20-556cb4da5ef4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '255cc0e9d9f748a19f8e104843259117'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:45:35.831395 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-d5bbcd06-1a4d-4de2-8d20-556cb4da5ef4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '255cc0e9d9f748a19f8e104843259117'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:45:35.841070 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-d5bbcd06-1a4d-4de2-8d20-556cb4da5ef4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '255cc0e9d9f748a19f8e104843259117'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:45:35.864160 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-d5bbcd06-1a4d-4de2-8d20-556cb4da5ef4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '255cc0e9d9f748a19f8e104843259117'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:45:35.872545 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1173/2377] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:45:35 2022] DELETE /identity/v3/users/255cc0e9d9f748a19f8e104843259117 => generated 0 bytes in 808 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:45:36.215645 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1cd82d0-d867-47dd-806f-5f7b61b6d216 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:45:36.217225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-b1cd82d0-d867-47dd-806f-5f7b61b6d216 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:45:36.232838 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1cd82d0-d867-47dd-806f-5f7b61b6d216 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:45:36.300694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82f3bdc-47e7-485b-a137-d8f9613ae799 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:45:36.301747 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82f3bdc-47e7-485b-a137-d8f9613ae799 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:36.302114 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82f3bdc-47e7-485b-a137-d8f9613ae799 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:36.302479 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82f3bdc-47e7-485b-a137-d8f9613ae799 admin admin] PATH_INFO: `/v3/projects/43ac40785a3e4fd7990f9edfd11b2f12` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:36.304992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82f3bdc-47e7-485b-a137-d8f9613ae799 admin admin] RBAC: Authorizing `identity:delete_project(project_id=43ac40785a3e4fd7990f9edfd11b2f12)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:45:36.308276 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82f3bdc-47e7-485b-a137-d8f9613ae799 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:45:36.426064 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:45:36.437383 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-f82f3bdc-47e7-485b-a137-d8f9613ae799 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '43ac40785a3e4fd7990f9edfd11b2f12', 'request_id': 'req-f82f3bdc-47e7-485b-a137-d8f9613ae799'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:45:36.446160 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-f82f3bdc-47e7-485b-a137-d8f9613ae799 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '43ac40785a3e4fd7990f9edfd11b2f12', 'request_id': 'req-f82f3bdc-47e7-485b-a137-d8f9613ae799'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:45:36.545145 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-f82f3bdc-47e7-485b-a137-d8f9613ae799 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '43ac40785a3e4fd7990f9edfd11b2f12', 'request_id': 'req-f82f3bdc-47e7-485b-a137-d8f9613ae799'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:45:36.560868 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-f82f3bdc-47e7-485b-a137-d8f9613ae799 admin admin] The token cache is being invalidate because project 43ac40785a3e4fd7990f9edfd11b2f12 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:45:36.561458 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-f82f3bdc-47e7-485b-a137-d8f9613ae799 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:45:36.562167 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-f82f3bdc-47e7-485b-a137-d8f9613ae799 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:45:36.564567 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1205/2378] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:45:36 2022] DELETE /identity/v3/projects/43ac40785a3e4fd7990f9edfd11b2f12 => generated 0 bytes in 349 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:45:46.430162 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=35,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:45:46.636558 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:DELETE=5,keystone:SELECT=11,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:45:49.306111 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5bbcd06-1a4d-4de2-8d20-556cb4da5ef4 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:45:49.307774 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-d5bbcd06-1a4d-4de2-8d20-556cb4da5ef4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:45:49.367590 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5bbcd06-1a4d-4de2-8d20-556cb4da5ef4 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:45:49.417811 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66aa5f2c-b6e7-44bb-ad83-741a3a2b3902 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:45:49.418592 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66aa5f2c-b6e7-44bb-ad83-741a3a2b3902 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:49.418888 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66aa5f2c-b6e7-44bb-ad83-741a3a2b3902 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:49.419175 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66aa5f2c-b6e7-44bb-ad83-741a3a2b3902 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:49.421018 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-66aa5f2c-b6e7-44bb-ad83-741a3a2b3902 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:45:49.431610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66aa5f2c-b6e7-44bb-ad83-741a3a2b3902 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:45:49.433083 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66aa5f2c-b6e7-44bb-ad83-741a3a2b3902 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:45:49.441323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1174/2379] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:45:49 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:45:49.771334 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947cc08f-4b61-4e95-998b-3a842baf64fc None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:49.771878 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947cc08f-4b61-4e95-998b-3a842baf64fc None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:49.772282 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947cc08f-4b61-4e95-998b-3a842baf64fc None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:49.774803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1206/2380] 10.222.0.16 () {54 vars in 921 bytes} [Tue Jul 12 17:45:49 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:45:49.781185 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a0a310-578c-4b5e-bfa6-2f057aada81a None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:49.781491 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a0a310-578c-4b5e-bfa6-2f057aada81a None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:49.781786 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a0a310-578c-4b5e-bfa6-2f057aada81a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:49.804159 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-39a0a310-578c-4b5e-bfa6-2f057aada81a None None] MFA Rules not processed for user `9012d5b88f554dc596582a7693c2a72a`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:45:49.805457 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-39a0a310-578c-4b5e-bfa6-2f057aada81a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:45:49.878127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1175/2381] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Jul 12 17:45:49 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:45:49.890245 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947cc08f-4b61-4e95-998b-3a842baf64fc None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:45:49.892142 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-947cc08f-4b61-4e95-998b-3a842baf64fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:45:49.953732 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947cc08f-4b61-4e95-998b-3a842baf64fc None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:45:49.957645 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa6a58d-c8d3-4921-a573-3afaf4e954b0 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4018d94a1feb4d6dade88ee9663f9729', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:45:49.958307 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa6a58d-c8d3-4921-a573-3afaf4e954b0 service glance] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:49.958626 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa6a58d-c8d3-4921-a573-3afaf4e954b0 service glance] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:49.958912 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa6a58d-c8d3-4921-a573-3afaf4e954b0 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:49.970616 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa6a58d-c8d3-4921-a573-3afaf4e954b0 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:45:49.972134 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fa6a58d-c8d3-4921-a573-3afaf4e954b0 service glance] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:45:49.984330 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1207/2382] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:45:49 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:45:50.189481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29188c4f-76f8-4157-a5d2-77c942c3e624 None None] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:50.189481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29188c4f-76f8-4157-a5d2-77c942c3e624 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:50.189481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29188c4f-76f8-4157-a5d2-77c942c3e624 None None] PATH_INFO: `/` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:50.189481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1176/2383] 10.222.0.16 () {54 vars in 921 bytes} [Tue Jul 12 17:45:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:45:50.195372 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5d92a2-24cc-44c3-b3ed-ffdb033a5c77 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:50.195665 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5d92a2-24cc-44c3-b3ed-ffdb033a5c77 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:50.196884 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5d92a2-24cc-44c3-b3ed-ffdb033a5c77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:50.219219 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-0f5d92a2-24cc-44c3-b3ed-ffdb033a5c77 None None] MFA Rules not processed for user `bf5bf02ae13d4d86a1b157369ef07a00`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:45:50.220427 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-0f5d92a2-24cc-44c3-b3ed-ffdb033a5c77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:45:50.227060 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1208/2384] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Jul 12 17:45:50 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:45:50.238868 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29188c4f-76f8-4157-a5d2-77c942c3e624 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:45:50.240609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-29188c4f-76f8-4157-a5d2-77c942c3e624 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:45:50.254756 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29188c4f-76f8-4157-a5d2-77c942c3e624 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:45:50.258358 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582fce71-a899-449f-b680-b7637fb8c422 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:45:50.259576 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582fce71-a899-449f-b680-b7637fb8c422 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:50.259867 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582fce71-a899-449f-b680-b7637fb8c422 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:50.260150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582fce71-a899-449f-b680-b7637fb8c422 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:50.267905 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-582fce71-a899-449f-b680-b7637fb8c422 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:45:50.269349 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-582fce71-a899-449f-b680-b7637fb8c422 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:45:50.278546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1177/2385] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:45:50 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:45:50.361324 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7026039-5b01-4c23-9fd4-676cfc4c47d4 None None] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:50.361615 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7026039-5b01-4c23-9fd4-676cfc4c47d4 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:50.361901 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7026039-5b01-4c23-9fd4-676cfc4c47d4 None None] PATH_INFO: `/` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:50.363335 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1209/2386] 10.222.0.16 () {54 vars in 921 bytes} [Tue Jul 12 17:45:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 12 17:45:50.369909 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7d4dff-fba2-4434-b3c4-c3bd7b7d17c1 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:50.370502 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7d4dff-fba2-4434-b3c4-c3bd7b7d17c1 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:50.370502 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7d4dff-fba2-4434-b3c4-c3bd7b7d17c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:50.389805 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-bc7d4dff-fba2-4434-b3c4-c3bd7b7d17c1 None None] MFA Rules not processed for user `9012d5b88f554dc596582a7693c2a72a`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:45:50.391069 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-bc7d4dff-fba2-4434-b3c4-c3bd7b7d17c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:45:50.401593 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1178/2387] 10.222.0.16 () {58 vars in 1018 bytes} [Tue Jul 12 17:45:50 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:45:50.412648 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7026039-5b01-4c23-9fd4-676cfc4c47d4 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:45:50.415024 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-d7026039-5b01-4c23-9fd4-676cfc4c47d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:45:50.429411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7026039-5b01-4c23-9fd4-676cfc4c47d4 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:45:50.434274 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42bc5289-dfe3-44a4-9e68-5edaec17c33b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:45:50.434975 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42bc5289-dfe3-44a4-9e68-5edaec17c33b service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:50.435262 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42bc5289-dfe3-44a4-9e68-5edaec17c33b service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:50.435538 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42bc5289-dfe3-44a4-9e68-5edaec17c33b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:50.446786 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42bc5289-dfe3-44a4-9e68-5edaec17c33b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:45:50.448215 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42bc5289-dfe3-44a4-9e68-5edaec17c33b service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:45:50.459649 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1210/2388] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:45:50 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:45:52.817289 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c06f5fb-4044-4bfe-ab84-ff013d8d585a None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:52.817846 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c06f5fb-4044-4bfe-ab84-ff013d8d585a None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:52.818041 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c06f5fb-4044-4bfe-ab84-ff013d8d585a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:52.838614 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-8c06f5fb-4044-4bfe-ab84-ff013d8d585a None None] MFA Rules not processed for user `3ac0d5d8de4943a3888abee3b24b8a6f`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:45:52.839872 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-8c06f5fb-4044-4bfe-ab84-ff013d8d585a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:45:52.874813 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:45:52.907138 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1179/2389] 10.222.0.16 () {58 vars in 1009 bytes} [Tue Jul 12 17:45:52 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:45:52.919725 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42bc5289-dfe3-44a4-9e68-5edaec17c33b service neutron] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:45:52.921433 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-42bc5289-dfe3-44a4-9e68-5edaec17c33b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:45:53.002257 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42bc5289-dfe3-44a4-9e68-5edaec17c33b service neutron] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:45:53.007805 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdfe7a5e-7046-49d8-b168-c9580cb38d93 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7906d1a19c642bd93507a27bf3e3dae', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:45:53.009171 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdfe7a5e-7046-49d8-b168-c9580cb38d93 service swift] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:45:53.009458 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdfe7a5e-7046-49d8-b168-c9580cb38d93 service swift] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:45:53.009750 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdfe7a5e-7046-49d8-b168-c9580cb38d93 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:45:53.019002 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdfe7a5e-7046-49d8-b168-c9580cb38d93 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:45:53.020690 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdfe7a5e-7046-49d8-b168-c9580cb38d93 service swift] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:45:53.029944 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1211/2390] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Jul 12 17:45:52 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 12 17:46:02.910382 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=11 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:46:03.026530 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=67 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:46:09.321968 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864281b5-4511-475d-a7a6-aa0b48b22b1d None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:09.322611 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864281b5-4511-475d-a7a6-aa0b48b22b1d None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:09.322908 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864281b5-4511-475d-a7a6-aa0b48b22b1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:09.341324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-864281b5-4511-475d-a7a6-aa0b48b22b1d None None] MFA Rules not processed for user `3ac0d5d8de4943a3888abee3b24b8a6f`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:46:09.342967 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-864281b5-4511-475d-a7a6-aa0b48b22b1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:46:09.350842 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1180/2391] 10.222.0.16 () {58 vars in 967 bytes} [Tue Jul 12 17:46:09 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:46:09.364039 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdfe7a5e-7046-49d8-b168-c9580cb38d93 service swift] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:09.375372 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdfe7a5e-7046-49d8-b168-c9580cb38d93 service swift] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:09.380455 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9776b75-5b4d-442d-814c-e13730eafb6b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd7906d1a19c642bd93507a27bf3e3dae', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:09.381087 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9776b75-5b4d-442d-814c-e13730eafb6b service swift] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:09.381374 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9776b75-5b4d-442d-814c-e13730eafb6b service swift] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:09.381652 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9776b75-5b4d-442d-814c-e13730eafb6b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:09.390343 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9776b75-5b4d-442d-814c-e13730eafb6b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:09.391943 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9776b75-5b4d-442d-814c-e13730eafb6b service swift] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:09.400145 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1212/2392] 10.222.0.16 () {60 vars in 1384 bytes} [Tue Jul 12 17:46:09 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 12 17:46:18.610499 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864281b5-4511-475d-a7a6-aa0b48b22b1d None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:18.620435 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864281b5-4511-475d-a7a6-aa0b48b22b1d None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:18.625234 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a5b881-740e-4731-b776-ff5916f5e089 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:18.625890 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a5b881-740e-4731-b776-ff5916f5e089 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:18.626478 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a5b881-740e-4731-b776-ff5916f5e089 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:18.626478 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a5b881-740e-4731-b776-ff5916f5e089 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:18.628722 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-39a5b881-740e-4731-b776-ff5916f5e089 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:46:18.695257 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a5b881-740e-4731-b776-ff5916f5e089 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:18.696826 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a5b881-740e-4731-b776-ff5916f5e089 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:18.706761 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1181/2393] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:46:18 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:46:19.396820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=6 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:46:22.112050 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9776b75-5b4d-442d-814c-e13730eafb6b service swift] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:22.113802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-f9776b75-5b4d-442d-814c-e13730eafb6b service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:46:22.128278 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9776b75-5b4d-442d-814c-e13730eafb6b service swift] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:22.133034 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988222dc-aa3c-4132-88a4-b1798a2205ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:22.133907 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988222dc-aa3c-4132-88a4-b1798a2205ce admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:22.134195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988222dc-aa3c-4132-88a4-b1798a2205ce admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:22.134505 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988222dc-aa3c-4132-88a4-b1798a2205ce admin admin] PATH_INFO: `/v3/users/c8441a4a72154ee3bbb95af66abf2ffd` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:22.135741 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988222dc-aa3c-4132-88a4-b1798a2205ce admin admin] RBAC: Authorizing `identity:delete_user(user_id=c8441a4a72154ee3bbb95af66abf2ffd)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:22.137748 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988222dc-aa3c-4132-88a4-b1798a2205ce admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:22.365218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-988222dc-aa3c-4132-88a4-b1798a2205ce admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c8441a4a72154ee3bbb95af66abf2ffd'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:22.365845 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-988222dc-aa3c-4132-88a4-b1798a2205ce admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c8441a4a72154ee3bbb95af66abf2ffd'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:22.366489 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-988222dc-aa3c-4132-88a4-b1798a2205ce admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c8441a4a72154ee3bbb95af66abf2ffd'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:22.373721 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-988222dc-aa3c-4132-88a4-b1798a2205ce admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c8441a4a72154ee3bbb95af66abf2ffd'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:22.386003 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-988222dc-aa3c-4132-88a4-b1798a2205ce admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c8441a4a72154ee3bbb95af66abf2ffd'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:22.488502 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1213/2394] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:46:22 2022] DELETE /identity/v3/users/c8441a4a72154ee3bbb95af66abf2ffd => generated 0 bytes in 377 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:46:22.975180 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a5b881-740e-4731-b776-ff5916f5e089 service neutron] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:22.986093 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a5b881-740e-4731-b776-ff5916f5e089 service neutron] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:23.045250 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1666eaf4-b1b4-4174-ab96-b667051ad074 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:23.046239 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1666eaf4-b1b4-4174-ab96-b667051ad074 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:23.046556 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1666eaf4-b1b4-4174-ab96-b667051ad074 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:23.046842 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1666eaf4-b1b4-4174-ab96-b667051ad074 admin admin] PATH_INFO: `/v3/projects/84b840eaa59d4f73ac68926437f03d82` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:23.048023 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1666eaf4-b1b4-4174-ab96-b667051ad074 admin admin] RBAC: Authorizing `identity:delete_project(project_id=84b840eaa59d4f73ac68926437f03d82)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:23.049944 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1666eaf4-b1b4-4174-ab96-b667051ad074 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:23.198457 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-1666eaf4-b1b4-4174-ab96-b667051ad074 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '84b840eaa59d4f73ac68926437f03d82', 'request_id': 'req-1666eaf4-b1b4-4174-ab96-b667051ad074'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:23.205547 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-1666eaf4-b1b4-4174-ab96-b667051ad074 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '84b840eaa59d4f73ac68926437f03d82', 'request_id': 'req-1666eaf4-b1b4-4174-ab96-b667051ad074'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:23.211272 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-1666eaf4-b1b4-4174-ab96-b667051ad074 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '84b840eaa59d4f73ac68926437f03d82', 'request_id': 'req-1666eaf4-b1b4-4174-ab96-b667051ad074'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:23.304874 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-1666eaf4-b1b4-4174-ab96-b667051ad074 admin admin] The token cache is being invalidate because project 84b840eaa59d4f73ac68926437f03d82 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:46:23.305167 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-1666eaf4-b1b4-4174-ab96-b667051ad074 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:23.305827 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-1666eaf4-b1b4-4174-ab96-b667051ad074 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:23.307800 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1182/2395] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:46:22 2022] DELETE /identity/v3/projects/84b840eaa59d4f73ac68926437f03d82 => generated 0 bytes in 333 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:46:23.313408 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4826eb22-edd0-4711-9f4d-a1e04fc32ab4 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:23.313990 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4826eb22-edd0-4711-9f4d-a1e04fc32ab4 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:23.320998 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4826eb22-edd0-4711-9f4d-a1e04fc32ab4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:23.333165 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-4826eb22-edd0-4711-9f4d-a1e04fc32ab4 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:46:23.334557 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-4826eb22-edd0-4711-9f4d-a1e04fc32ab4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:46:23.396873 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1214/2396] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:46:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:46:23.404731 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89db43ee-3a8f-4475-aed7-fd286cc036aa None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:23.405032 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89db43ee-3a8f-4475-aed7-fd286cc036aa None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:23.405324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89db43ee-3a8f-4475-aed7-fd286cc036aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:23.424674 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-89db43ee-3a8f-4475-aed7-fd286cc036aa None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:46:23.425913 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-89db43ee-3a8f-4475-aed7-fd286cc036aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:46:23.434113 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1183/2397] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:46:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:46:23.439829 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4826eb22-edd0-4711-9f4d-a1e04fc32ab4 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:23.457836 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4826eb22-edd0-4711-9f4d-a1e04fc32ab4 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:23.465203 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bdf945c-28a1-4a76-ab6b-3b0b119ad0f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:23.466262 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bdf945c-28a1-4a76-ab6b-3b0b119ad0f1 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:23.470100 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bdf945c-28a1-4a76-ab6b-3b0b119ad0f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:23.470100 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bdf945c-28a1-4a76-ab6b-3b0b119ad0f1 admin admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:23.470100 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bdf945c-28a1-4a76-ab6b-3b0b119ad0f1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:46:23.470100 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bdf945c-28a1-4a76-ab6b-3b0b119ad0f1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:23.472489 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bdf945c-28a1-4a76-ab6b-3b0b119ad0f1 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:23.490150 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1215/2398] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:46:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:46:23.494447 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89db43ee-3a8f-4475-aed7-fd286cc036aa None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:23.504406 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89db43ee-3a8f-4475-aed7-fd286cc036aa None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:23.509476 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125c46f0-9a01-4aba-b6b3-6fbc41e27835 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:23.510172 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125c46f0-9a01-4aba-b6b3-6fbc41e27835 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:23.510499 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125c46f0-9a01-4aba-b6b3-6fbc41e27835 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:23.510787 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-125c46f0-9a01-4aba-b6b3-6fbc41e27835 admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:23.511501 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-125c46f0-9a01-4aba-b6b3-6fbc41e27835 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:23.513398 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-125c46f0-9a01-4aba-b6b3-6fbc41e27835 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:23.522032 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:46:23.609891 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1184/2399] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:46:23 2022] POST /identity/v3/projects => generated 363 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:46:23.614287 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bdf945c-28a1-4a76-ab6b-3b0b119ad0f1 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:23.625311 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bdf945c-28a1-4a76-ab6b-3b0b119ad0f1 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:23.686670 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee97e57e-1f04-4cd8-bad3-90c93c0a4554 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:23.691194 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee97e57e-1f04-4cd8-bad3-90c93c0a4554 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:23.691194 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee97e57e-1f04-4cd8-bad3-90c93c0a4554 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:23.691194 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee97e57e-1f04-4cd8-bad3-90c93c0a4554 admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:23.691194 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee97e57e-1f04-4cd8-bad3-90c93c0a4554 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:23.691194 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee97e57e-1f04-4cd8-bad3-90c93c0a4554 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:23.697558 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:46:23.791550 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1216/2400] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:46:23 2022] POST /identity/v3/users => generated 282 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:46:23.795983 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125c46f0-9a01-4aba-b6b3-6fbc41e27835 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:23.806595 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-125c46f0-9a01-4aba-b6b3-6fbc41e27835 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:23.811834 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e514d7d5-ca29-42ac-b40a-dd2e9c98d1ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:23.812647 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e514d7d5-ca29-42ac-b40a-dd2e9c98d1ec admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:23.812937 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e514d7d5-ca29-42ac-b40a-dd2e9c98d1ec admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:23.813219 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e514d7d5-ca29-42ac-b40a-dd2e9c98d1ec admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:23.813965 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e514d7d5-ca29-42ac-b40a-dd2e9c98d1ec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:23.815601 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e514d7d5-ca29-42ac-b40a-dd2e9c98d1ec admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:23.827801 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1185/2401] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:46:23 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:46:23.833156 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee97e57e-1f04-4cd8-bad3-90c93c0a4554 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:23.843887 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee97e57e-1f04-4cd8-bad3-90c93c0a4554 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:23.849447 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232b75a8-5b43-4924-bc84-39a3a4342b27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:23.850418 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232b75a8-5b43-4924-bc84-39a3a4342b27 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:23.850814 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232b75a8-5b43-4924-bc84-39a3a4342b27 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:23.854566 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232b75a8-5b43-4924-bc84-39a3a4342b27 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:23.854566 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-232b75a8-5b43-4924-bc84-39a3a4342b27 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:23.854566 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-232b75a8-5b43-4924-bc84-39a3a4342b27 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:23.865885 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1217/2402] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:46:23 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:46:23.870224 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e514d7d5-ca29-42ac-b40a-dd2e9c98d1ec admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:23.881233 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e514d7d5-ca29-42ac-b40a-dd2e9c98d1ec admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:23.886033 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804eadf1-fb36-47a1-997e-7a5be2a08114 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:23.886999 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804eadf1-fb36-47a1-997e-7a5be2a08114 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:23.887288 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804eadf1-fb36-47a1-997e-7a5be2a08114 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:23.887569 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804eadf1-fb36-47a1-997e-7a5be2a08114 admin admin] PATH_INFO: `/v3/projects/b491afa5793b411fad60e5257fb6d018/users/cbf27ba30b6442e8a999ed270c58d7ba/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:23.904045 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804eadf1-fb36-47a1-997e-7a5be2a08114 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b491afa5793b411fad60e5257fb6d018, user_id=cbf27ba30b6442e8a999ed270c58d7ba, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:23.906922 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804eadf1-fb36-47a1-997e-7a5be2a08114 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:24.008484 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1186/2403] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:46:23 2022] PUT /identity/v3/projects/b491afa5793b411fad60e5257fb6d018/users/cbf27ba30b6442e8a999ed270c58d7ba/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:46:24.020488 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b44b10-1916-4cdd-8ca4-147f4fa4ca77 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:24.020781 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b44b10-1916-4cdd-8ca4-147f4fa4ca77 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:24.021061 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b44b10-1916-4cdd-8ca4-147f4fa4ca77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:24.049142 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-a6b44b10-1916-4cdd-8ca4-147f4fa4ca77 None None] MFA Rules not processed for user `cbf27ba30b6442e8a999ed270c58d7ba`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:46:24.050383 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-a6b44b10-1916-4cdd-8ca4-147f4fa4ca77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:46:24.115624 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1218/2404] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:46:24 2022] POST /identity/v3/auth/tokens => generated 3230 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:46:24.119994 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804eadf1-fb36-47a1-997e-7a5be2a08114 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:24.183507 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804eadf1-fb36-47a1-997e-7a5be2a08114 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:24.188507 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c93ba13-c064-433d-8df8-42664354808f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:24.189150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c93ba13-c064-433d-8df8-42664354808f admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:24.189435 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c93ba13-c064-433d-8df8-42664354808f admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:24.189714 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c93ba13-c064-433d-8df8-42664354808f admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:24.190450 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c93ba13-c064-433d-8df8-42664354808f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:24.192355 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c93ba13-c064-433d-8df8-42664354808f admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:24.255596 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1187/2405] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:46:24 2022] POST /identity/v3/projects => generated 363 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:46:24.259886 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b44b10-1916-4cdd-8ca4-147f4fa4ca77 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:24.272409 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b44b10-1916-4cdd-8ca4-147f4fa4ca77 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:24.330865 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5271f96-7833-40a0-973d-b89ba28dc989 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:24.331703 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5271f96-7833-40a0-973d-b89ba28dc989 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:24.332034 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5271f96-7833-40a0-973d-b89ba28dc989 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:24.332299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5271f96-7833-40a0-973d-b89ba28dc989 admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:24.333007 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5271f96-7833-40a0-973d-b89ba28dc989 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:24.334962 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5271f96-7833-40a0-973d-b89ba28dc989 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:24.446341 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1219/2406] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:46:24 2022] POST /identity/v3/users => generated 288 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:46:24.447554 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c93ba13-c064-433d-8df8-42664354808f admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:24.459290 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c93ba13-c064-433d-8df8-42664354808f admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:24.464370 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5552d5b9-6384-43ea-8a1e-724a38d0195b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:24.470572 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5552d5b9-6384-43ea-8a1e-724a38d0195b admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:24.470572 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5552d5b9-6384-43ea-8a1e-724a38d0195b admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:24.470572 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5552d5b9-6384-43ea-8a1e-724a38d0195b admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:24.470572 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5552d5b9-6384-43ea-8a1e-724a38d0195b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:24.470572 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5552d5b9-6384-43ea-8a1e-724a38d0195b admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:24.479537 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1188/2407] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:46:24 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:46:24.483691 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5271f96-7833-40a0-973d-b89ba28dc989 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:24.494175 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5271f96-7833-40a0-973d-b89ba28dc989 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:24.498876 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4935012-fed9-4d5a-a3c8-b026c419151c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:24.499761 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4935012-fed9-4d5a-a3c8-b026c419151c admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:24.500052 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4935012-fed9-4d5a-a3c8-b026c419151c admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:24.500336 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4935012-fed9-4d5a-a3c8-b026c419151c admin admin] PATH_INFO: `/v3/projects/78cc9d12c71a4dc980ae21a4950eb71c/users/da48529ce20c41159eb46de3abbb7607/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:24.517289 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4935012-fed9-4d5a-a3c8-b026c419151c admin admin] RBAC: Authorizing `identity:create_grant(project_id=78cc9d12c71a4dc980ae21a4950eb71c, user_id=da48529ce20c41159eb46de3abbb7607, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:24.520062 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4935012-fed9-4d5a-a3c8-b026c419151c admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:24.607288 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1220/2408] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:46:24 2022] PUT /identity/v3/projects/78cc9d12c71a4dc980ae21a4950eb71c/users/da48529ce20c41159eb46de3abbb7607/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:46:24.612026 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5552d5b9-6384-43ea-8a1e-724a38d0195b admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:24.622168 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5552d5b9-6384-43ea-8a1e-724a38d0195b admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:24.680802 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce62312a-0f56-4952-9501-ddc1ecd31208 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:24.681263 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce62312a-0f56-4952-9501-ddc1ecd31208 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:24.681550 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce62312a-0f56-4952-9501-ddc1ecd31208 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:24.681833 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce62312a-0f56-4952-9501-ddc1ecd31208 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:24.682597 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce62312a-0f56-4952-9501-ddc1ecd31208 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:24.684189 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce62312a-0f56-4952-9501-ddc1ecd31208 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:24.695383 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1189/2409] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:46:24 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:46:24.699901 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4935012-fed9-4d5a-a3c8-b026c419151c admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:24.711471 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4935012-fed9-4d5a-a3c8-b026c419151c admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:24.716375 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e2583e-5721-4800-8272-a688de1bc164 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:24.717253 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e2583e-5721-4800-8272-a688de1bc164 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:24.717539 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e2583e-5721-4800-8272-a688de1bc164 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:24.717814 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e2583e-5721-4800-8272-a688de1bc164 admin admin] PATH_INFO: `/v3/projects/78cc9d12c71a4dc980ae21a4950eb71c/users/da48529ce20c41159eb46de3abbb7607/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:24.719885 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e2583e-5721-4800-8272-a688de1bc164 admin admin] RBAC: Authorizing `identity:create_grant(project_id=78cc9d12c71a4dc980ae21a4950eb71c, user_id=da48529ce20c41159eb46de3abbb7607, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:24.722632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e2583e-5721-4800-8272-a688de1bc164 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:24.733533 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1221/2410] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:46:24 2022] PUT /identity/v3/projects/78cc9d12c71a4dc980ae21a4950eb71c/users/da48529ce20c41159eb46de3abbb7607/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:46:24.744254 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4d13f5-23ed-4fc3-9e63-dc54e6cbede5 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:24.744550 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4d13f5-23ed-4fc3-9e63-dc54e6cbede5 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:24.744834 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4d13f5-23ed-4fc3-9e63-dc54e6cbede5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:24.773039 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-6b4d13f5-23ed-4fc3-9e63-dc54e6cbede5 None None] MFA Rules not processed for user `da48529ce20c41159eb46de3abbb7607`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:46:24.774302 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-6b4d13f5-23ed-4fc3-9e63-dc54e6cbede5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:46:24.786595 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:46:24.847583 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1190/2411] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:46:24 2022] POST /identity/v3/auth/tokens => generated 3297 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:46:24.859293 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e2583e-5721-4800-8272-a688de1bc164 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:24.861743 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-66e2583e-5721-4800-8272-a688de1bc164 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:46:24.886123 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:INSERT=8,keystone:SELECT=92 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:46:24.924449 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e2583e-5721-4800-8272-a688de1bc164 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:24.928404 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-312d27b6-b905-4ba1-9684-53f1be7af9bf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:24.929058 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312d27b6-b905-4ba1-9684-53f1be7af9bf service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:24.929350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312d27b6-b905-4ba1-9684-53f1be7af9bf service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:24.929641 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312d27b6-b905-4ba1-9684-53f1be7af9bf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:24.942415 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-312d27b6-b905-4ba1-9684-53f1be7af9bf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:24.942415 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-312d27b6-b905-4ba1-9684-53f1be7af9bf service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:24.952129 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1222/2412] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:46:24 2022] GET /identity/v3/auth/tokens => generated 3297 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:46:25.842137 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4d13f5-23ed-4fc3-9e63-dc54e6cbede5 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:25.904892 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4d13f5-23ed-4fc3-9e63-dc54e6cbede5 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:25.910095 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddae26b5-6897-4621-9812-5c13e88063e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:25.910999 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddae26b5-6897-4621-9812-5c13e88063e8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:25.911284 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddae26b5-6897-4621-9812-5c13e88063e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:25.911561 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddae26b5-6897-4621-9812-5c13e88063e8 admin admin] PATH_INFO: `/v3/users/cbf27ba30b6442e8a999ed270c58d7ba` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:25.912752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddae26b5-6897-4621-9812-5c13e88063e8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=cbf27ba30b6442e8a999ed270c58d7ba)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:25.914704 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddae26b5-6897-4621-9812-5c13e88063e8 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:26.085769 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-ddae26b5-6897-4621-9812-5c13e88063e8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cbf27ba30b6442e8a999ed270c58d7ba'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:26.184922 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-ddae26b5-6897-4621-9812-5c13e88063e8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cbf27ba30b6442e8a999ed270c58d7ba'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:26.185529 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-ddae26b5-6897-4621-9812-5c13e88063e8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cbf27ba30b6442e8a999ed270c58d7ba'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:26.193421 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-ddae26b5-6897-4621-9812-5c13e88063e8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cbf27ba30b6442e8a999ed270c58d7ba'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:26.206796 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-ddae26b5-6897-4621-9812-5c13e88063e8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cbf27ba30b6442e8a999ed270c58d7ba'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:26.212884 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1191/2413] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:46:25 2022] DELETE /identity/v3/users/cbf27ba30b6442e8a999ed270c58d7ba => generated 0 bytes in 371 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:46:26.217374 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ee5000-3c73-49be-bc34-23e40d95a2c1 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:26.217862 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ee5000-3c73-49be-bc34-23e40d95a2c1 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:26.217987 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ee5000-3c73-49be-bc34-23e40d95a2c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:26.235810 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-23ee5000-3c73-49be-bc34-23e40d95a2c1 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:46:26.237140 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-23ee5000-3c73-49be-bc34-23e40d95a2c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:46:26.296125 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1223/2414] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:46:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:46:26.305769 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddae26b5-6897-4621-9812-5c13e88063e8 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:26.307191 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-ddae26b5-6897-4621-9812-5c13e88063e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:46:26.339952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:46:26.365345 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddae26b5-6897-4621-9812-5c13e88063e8 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:26.368945 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c5d5d7-8b05-4f97-8467-a2d63f509205 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:26.369591 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c5d5d7-8b05-4f97-8467-a2d63f509205 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:26.369877 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c5d5d7-8b05-4f97-8467-a2d63f509205 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:26.370156 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c5d5d7-8b05-4f97-8467-a2d63f509205 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:26.379551 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c5d5d7-8b05-4f97-8467-a2d63f509205 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:26.380988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c5d5d7-8b05-4f97-8467-a2d63f509205 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:26.391048 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1192/2415] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:46:26 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:46:26.797821 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ee5000-3c73-49be-bc34-23e40d95a2c1 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:26.799829 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-23ee5000-3c73-49be-bc34-23e40d95a2c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:46:26.812328 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ee5000-3c73-49be-bc34-23e40d95a2c1 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:26.817928 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96ffc28-9634-41bf-8cdc-b1eba4fb5534 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:26.818865 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96ffc28-9634-41bf-8cdc-b1eba4fb5534 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:26.819410 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96ffc28-9634-41bf-8cdc-b1eba4fb5534 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:26.819697 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96ffc28-9634-41bf-8cdc-b1eba4fb5534 admin admin] PATH_INFO: `/v3/projects/b491afa5793b411fad60e5257fb6d018` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:26.820899 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96ffc28-9634-41bf-8cdc-b1eba4fb5534 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b491afa5793b411fad60e5257fb6d018)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:26.822876 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96ffc28-9634-41bf-8cdc-b1eba4fb5534 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:26.952264 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-f96ffc28-9634-41bf-8cdc-b1eba4fb5534 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b491afa5793b411fad60e5257fb6d018', 'request_id': 'req-f96ffc28-9634-41bf-8cdc-b1eba4fb5534'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:26.957743 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-f96ffc28-9634-41bf-8cdc-b1eba4fb5534 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b491afa5793b411fad60e5257fb6d018', 'request_id': 'req-f96ffc28-9634-41bf-8cdc-b1eba4fb5534'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:27.024578 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-f96ffc28-9634-41bf-8cdc-b1eba4fb5534 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b491afa5793b411fad60e5257fb6d018', 'request_id': 'req-f96ffc28-9634-41bf-8cdc-b1eba4fb5534'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:27.036461 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-f96ffc28-9634-41bf-8cdc-b1eba4fb5534 admin admin] The token cache is being invalidate because project b491afa5793b411fad60e5257fb6d018 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:46:27.036754 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-f96ffc28-9634-41bf-8cdc-b1eba4fb5534 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:27.037399 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-f96ffc28-9634-41bf-8cdc-b1eba4fb5534 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:27.039010 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1224/2416] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:46:26 2022] DELETE /identity/v3/projects/b491afa5793b411fad60e5257fb6d018 => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:46:27.043210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c5d5d7-8b05-4f97-8467-a2d63f509205 service neutron] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:27.054066 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c5d5d7-8b05-4f97-8467-a2d63f509205 service neutron] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:27.110657 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f7d6b1-ba9b-4b5d-a834-243f79e5f154 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:27.112168 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f7d6b1-ba9b-4b5d-a834-243f79e5f154 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:27.112452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f7d6b1-ba9b-4b5d-a834-243f79e5f154 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:27.112732 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f7d6b1-ba9b-4b5d-a834-243f79e5f154 admin admin] PATH_INFO: `/v3/users/da48529ce20c41159eb46de3abbb7607` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:27.113920 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f7d6b1-ba9b-4b5d-a834-243f79e5f154 admin admin] RBAC: Authorizing `identity:delete_user(user_id=da48529ce20c41159eb46de3abbb7607)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:27.115958 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f7d6b1-ba9b-4b5d-a834-243f79e5f154 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:27.312221 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-e0f7d6b1-ba9b-4b5d-a834-243f79e5f154 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'da48529ce20c41159eb46de3abbb7607'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:27.384865 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-e0f7d6b1-ba9b-4b5d-a834-243f79e5f154 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'da48529ce20c41159eb46de3abbb7607'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:27.385488 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-e0f7d6b1-ba9b-4b5d-a834-243f79e5f154 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'da48529ce20c41159eb46de3abbb7607'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:27.393063 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-e0f7d6b1-ba9b-4b5d-a834-243f79e5f154 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'da48529ce20c41159eb46de3abbb7607'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:27.405029 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-e0f7d6b1-ba9b-4b5d-a834-243f79e5f154 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'da48529ce20c41159eb46de3abbb7607'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:27.411596 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1193/2417] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:46:27 2022] DELETE /identity/v3/users/da48529ce20c41159eb46de3abbb7607 => generated 0 bytes in 369 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:46:27.889515 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96ffc28-9634-41bf-8cdc-b1eba4fb5534 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:27.891192 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-f96ffc28-9634-41bf-8cdc-b1eba4fb5534 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:46:27.904812 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96ffc28-9634-41bf-8cdc-b1eba4fb5534 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:27.952196 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:46:27.964680 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c72275a-461d-4d77-9ff2-c30cd8085f28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:27.965596 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c72275a-461d-4d77-9ff2-c30cd8085f28 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:27.965887 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c72275a-461d-4d77-9ff2-c30cd8085f28 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:27.966172 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c72275a-461d-4d77-9ff2-c30cd8085f28 admin admin] PATH_INFO: `/v3/projects/78cc9d12c71a4dc980ae21a4950eb71c` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:27.967394 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c72275a-461d-4d77-9ff2-c30cd8085f28 admin admin] RBAC: Authorizing `identity:delete_project(project_id=78cc9d12c71a4dc980ae21a4950eb71c)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:27.969318 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c72275a-461d-4d77-9ff2-c30cd8085f28 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:28.082500 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-9c72275a-461d-4d77-9ff2-c30cd8085f28 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '78cc9d12c71a4dc980ae21a4950eb71c', 'request_id': 'req-9c72275a-461d-4d77-9ff2-c30cd8085f28'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:28.087867 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-9c72275a-461d-4d77-9ff2-c30cd8085f28 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '78cc9d12c71a4dc980ae21a4950eb71c', 'request_id': 'req-9c72275a-461d-4d77-9ff2-c30cd8085f28'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:28.179100 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-9c72275a-461d-4d77-9ff2-c30cd8085f28 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '78cc9d12c71a4dc980ae21a4950eb71c', 'request_id': 'req-9c72275a-461d-4d77-9ff2-c30cd8085f28'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:28.191646 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-9c72275a-461d-4d77-9ff2-c30cd8085f28 admin admin] The token cache is being invalidate because project 78cc9d12c71a4dc980ae21a4950eb71c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:46:28.192256 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-9c72275a-461d-4d77-9ff2-c30cd8085f28 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:28.193589 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-9c72275a-461d-4d77-9ff2-c30cd8085f28 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:28.195868 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1225/2418] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:46:27 2022] DELETE /identity/v3/projects/78cc9d12c71a4dc980ae21a4950eb71c => generated 0 bytes in 307 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:46:28.200698 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3634c187-4284-493a-9691-1e3eb229e6de None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:28.201148 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3634c187-4284-493a-9691-1e3eb229e6de None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:28.201268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3634c187-4284-493a-9691-1e3eb229e6de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:28.219946 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-3634c187-4284-493a-9691-1e3eb229e6de None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:46:28.221274 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-3634c187-4284-493a-9691-1e3eb229e6de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:46:28.265644 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:46:28.280579 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1194/2419] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:46:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:46:28.289701 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490fbc8c-b6ba-4511-ac20-f52bb2b62c68 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:28.289991 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490fbc8c-b6ba-4511-ac20-f52bb2b62c68 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:28.290269 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490fbc8c-b6ba-4511-ac20-f52bb2b62c68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:28.309879 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-490fbc8c-b6ba-4511-ac20-f52bb2b62c68 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:46:28.311198 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-490fbc8c-b6ba-4511-ac20-f52bb2b62c68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:46:28.320455 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1226/2420] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:46:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:46:28.324809 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3634c187-4284-493a-9691-1e3eb229e6de None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:28.335123 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3634c187-4284-493a-9691-1e3eb229e6de None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:28.343608 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695adfec-31cc-43cd-b66e-4faae3c14bd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:28.343608 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695adfec-31cc-43cd-b66e-4faae3c14bd6 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:28.343608 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695adfec-31cc-43cd-b66e-4faae3c14bd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:28.343608 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695adfec-31cc-43cd-b66e-4faae3c14bd6 admin admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:28.343967 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-695adfec-31cc-43cd-b66e-4faae3c14bd6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:46:28.343967 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-695adfec-31cc-43cd-b66e-4faae3c14bd6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:28.344127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-695adfec-31cc-43cd-b66e-4faae3c14bd6 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:28.356648 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1195/2421] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:46:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:46:28.360888 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490fbc8c-b6ba-4511-ac20-f52bb2b62c68 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:28.371121 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490fbc8c-b6ba-4511-ac20-f52bb2b62c68 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:28.375774 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4d4cba-e568-4993-b677-6afb1fd9b2ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:28.376522 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4d4cba-e568-4993-b677-6afb1fd9b2ad admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:28.377459 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4d4cba-e568-4993-b677-6afb1fd9b2ad admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:28.377810 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4d4cba-e568-4993-b677-6afb1fd9b2ad admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:28.378497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4d4cba-e568-4993-b677-6afb1fd9b2ad admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:28.380365 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4d4cba-e568-4993-b677-6afb1fd9b2ad admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:28.490231 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1227/2422] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:46:28 2022] POST /identity/v3/projects => generated 363 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:46:28.496319 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695adfec-31cc-43cd-b66e-4faae3c14bd6 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:28.507151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695adfec-31cc-43cd-b66e-4faae3c14bd6 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:28.563480 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59fc8cfe-804f-469e-a125-009934c85b5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:28.564807 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fc8cfe-804f-469e-a125-009934c85b5b admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:28.565142 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fc8cfe-804f-469e-a125-009934c85b5b admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:28.565625 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fc8cfe-804f-469e-a125-009934c85b5b admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:28.566511 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59fc8cfe-804f-469e-a125-009934c85b5b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:28.568448 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59fc8cfe-804f-469e-a125-009934c85b5b admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:28.676577 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1196/2423] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:46:28 2022] POST /identity/v3/users => generated 282 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:46:28.680759 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4d4cba-e568-4993-b677-6afb1fd9b2ad admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:28.691858 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4d4cba-e568-4993-b677-6afb1fd9b2ad admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:28.696881 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6143b063-59a6-462c-b450-e11733514355 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:28.697548 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6143b063-59a6-462c-b450-e11733514355 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:28.697839 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6143b063-59a6-462c-b450-e11733514355 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:28.698123 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6143b063-59a6-462c-b450-e11733514355 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:28.698914 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6143b063-59a6-462c-b450-e11733514355 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:28.700530 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6143b063-59a6-462c-b450-e11733514355 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:28.711475 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1228/2424] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:46:28 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:46:28.715937 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59fc8cfe-804f-469e-a125-009934c85b5b admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:28.727286 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59fc8cfe-804f-469e-a125-009934c85b5b admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:28.731839 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7080ebd-77eb-4e3d-97ba-955ce48d7aa6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:28.732508 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7080ebd-77eb-4e3d-97ba-955ce48d7aa6 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:28.732796 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7080ebd-77eb-4e3d-97ba-955ce48d7aa6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:28.733080 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7080ebd-77eb-4e3d-97ba-955ce48d7aa6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:28.733813 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7080ebd-77eb-4e3d-97ba-955ce48d7aa6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:28.735444 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7080ebd-77eb-4e3d-97ba-955ce48d7aa6 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:28.746369 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1197/2425] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:46:28 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:46:28.750602 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6143b063-59a6-462c-b450-e11733514355 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:28.760969 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6143b063-59a6-462c-b450-e11733514355 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:28.765582 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6288dbbe-cf47-4848-a9a5-ccdf256ad38e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:28.766480 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6288dbbe-cf47-4848-a9a5-ccdf256ad38e admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:28.766772 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6288dbbe-cf47-4848-a9a5-ccdf256ad38e admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:28.767050 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6288dbbe-cf47-4848-a9a5-ccdf256ad38e admin admin] PATH_INFO: `/v3/projects/94676f7e0f51455a8cab7219eb2560e4/users/9bb12c28426f4137b66c7dc75cc38f28/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:28.781568 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6288dbbe-cf47-4848-a9a5-ccdf256ad38e admin admin] RBAC: Authorizing `identity:create_grant(project_id=94676f7e0f51455a8cab7219eb2560e4, user_id=9bb12c28426f4137b66c7dc75cc38f28, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:28.784854 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6288dbbe-cf47-4848-a9a5-ccdf256ad38e admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:28.881818 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1229/2426] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:46:28 2022] PUT /identity/v3/projects/94676f7e0f51455a8cab7219eb2560e4/users/9bb12c28426f4137b66c7dc75cc38f28/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:46:28.892489 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d2cd0d-1548-4c97-aa9e-5ac15446a4c3 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:28.892862 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d2cd0d-1548-4c97-aa9e-5ac15446a4c3 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:28.893136 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d2cd0d-1548-4c97-aa9e-5ac15446a4c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:28.921007 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-85d2cd0d-1548-4c97-aa9e-5ac15446a4c3 None None] MFA Rules not processed for user `9bb12c28426f4137b66c7dc75cc38f28`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:46:28.922233 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-85d2cd0d-1548-4c97-aa9e-5ac15446a4c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:46:28.988564 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1198/2427] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:46:28 2022] POST /identity/v3/auth/tokens => generated 3230 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:46:28.993031 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6288dbbe-cf47-4848-a9a5-ccdf256ad38e admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:29.054730 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6288dbbe-cf47-4848-a9a5-ccdf256ad38e admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:29.059581 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a4e2b6-b799-471c-a2c5-28ec1f9460c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:29.060241 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a4e2b6-b799-471c-a2c5-28ec1f9460c9 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:29.060532 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a4e2b6-b799-471c-a2c5-28ec1f9460c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:29.060816 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a4e2b6-b799-471c-a2c5-28ec1f9460c9 admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:29.061503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16a4e2b6-b799-471c-a2c5-28ec1f9460c9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:29.063446 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16a4e2b6-b799-471c-a2c5-28ec1f9460c9 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:29.139466 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:46:29.147195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1230/2428] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:46:28 2022] POST /identity/v3/projects => generated 363 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:46:29.151294 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d2cd0d-1548-4c97-aa9e-5ac15446a4c3 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:29.163041 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d2cd0d-1548-4c97-aa9e-5ac15446a4c3 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:29.208289 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:46:29.221517 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfee92e4-6dcc-4319-bcbf-bc293574a411 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:29.222203 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfee92e4-6dcc-4319-bcbf-bc293574a411 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:29.222517 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfee92e4-6dcc-4319-bcbf-bc293574a411 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:29.222803 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfee92e4-6dcc-4319-bcbf-bc293574a411 admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:29.223544 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfee92e4-6dcc-4319-bcbf-bc293574a411 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:29.225479 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfee92e4-6dcc-4319-bcbf-bc293574a411 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:29.339515 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1199/2429] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:46:29 2022] POST /identity/v3/users => generated 288 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:46:29.343824 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a4e2b6-b799-471c-a2c5-28ec1f9460c9 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:29.355418 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a4e2b6-b799-471c-a2c5-28ec1f9460c9 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:29.360328 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77712ea3-ded5-46a3-b6f2-cfdf1b1add03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:29.361030 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77712ea3-ded5-46a3-b6f2-cfdf1b1add03 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:29.361331 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77712ea3-ded5-46a3-b6f2-cfdf1b1add03 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:29.361613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77712ea3-ded5-46a3-b6f2-cfdf1b1add03 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:29.362358 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77712ea3-ded5-46a3-b6f2-cfdf1b1add03 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:29.363991 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77712ea3-ded5-46a3-b6f2-cfdf1b1add03 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:29.374714 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1231/2430] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:46:29 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:46:29.379117 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfee92e4-6dcc-4319-bcbf-bc293574a411 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:29.389369 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfee92e4-6dcc-4319-bcbf-bc293574a411 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:29.394739 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b4c920-fefd-473e-8782-30abaa8dee1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:29.395646 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b4c920-fefd-473e-8782-30abaa8dee1a admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:29.401500 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b4c920-fefd-473e-8782-30abaa8dee1a admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:29.401500 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b4c920-fefd-473e-8782-30abaa8dee1a admin admin] PATH_INFO: `/v3/projects/3984ab8f799544ae88219d2cbb87fe95/users/ca881c33822a4efb9fcb0cb1856572c6/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:29.411571 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5b4c920-fefd-473e-8782-30abaa8dee1a admin admin] RBAC: Authorizing `identity:create_grant(project_id=3984ab8f799544ae88219d2cbb87fe95, user_id=ca881c33822a4efb9fcb0cb1856572c6, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:29.414369 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5b4c920-fefd-473e-8782-30abaa8dee1a admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:29.482163 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1200/2431] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:46:29 2022] PUT /identity/v3/projects/3984ab8f799544ae88219d2cbb87fe95/users/ca881c33822a4efb9fcb0cb1856572c6/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:46:29.486366 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77712ea3-ded5-46a3-b6f2-cfdf1b1add03 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:29.496633 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77712ea3-ded5-46a3-b6f2-cfdf1b1add03 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:29.555712 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217688e5-4d96-4331-a9a4-e3a34d021fc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:29.556378 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217688e5-4d96-4331-a9a4-e3a34d021fc1 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:29.556664 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217688e5-4d96-4331-a9a4-e3a34d021fc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:29.556942 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217688e5-4d96-4331-a9a4-e3a34d021fc1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:29.557673 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217688e5-4d96-4331-a9a4-e3a34d021fc1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:29.559299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217688e5-4d96-4331-a9a4-e3a34d021fc1 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:29.573726 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1232/2432] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:46:29 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:46:29.578218 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b4c920-fefd-473e-8782-30abaa8dee1a admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:29.586609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b4c920-fefd-473e-8782-30abaa8dee1a admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:29.591424 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8714752e-3284-49fa-a99f-7c950fd26755 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:29.592328 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8714752e-3284-49fa-a99f-7c950fd26755 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:29.592624 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8714752e-3284-49fa-a99f-7c950fd26755 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:29.592961 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8714752e-3284-49fa-a99f-7c950fd26755 admin admin] PATH_INFO: `/v3/projects/3984ab8f799544ae88219d2cbb87fe95/users/ca881c33822a4efb9fcb0cb1856572c6/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:29.595139 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8714752e-3284-49fa-a99f-7c950fd26755 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3984ab8f799544ae88219d2cbb87fe95, user_id=ca881c33822a4efb9fcb0cb1856572c6, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:29.597959 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8714752e-3284-49fa-a99f-7c950fd26755 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:29.609006 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1201/2433] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:46:29 2022] PUT /identity/v3/projects/3984ab8f799544ae88219d2cbb87fe95/users/ca881c33822a4efb9fcb0cb1856572c6/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:46:29.619410 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46587cdb-5638-468e-b09e-1e301000beb2 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:29.619835 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46587cdb-5638-468e-b09e-1e301000beb2 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:29.620194 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46587cdb-5638-468e-b09e-1e301000beb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:29.650001 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-46587cdb-5638-468e-b09e-1e301000beb2 None None] MFA Rules not processed for user `ca881c33822a4efb9fcb0cb1856572c6`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:46:29.651436 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-46587cdb-5638-468e-b09e-1e301000beb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:46:29.722548 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1233/2434] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:46:29 2022] POST /identity/v3/auth/tokens => generated 3297 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:46:29.733669 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8714752e-3284-49fa-a99f-7c950fd26755 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:29.735505 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-8714752e-3284-49fa-a99f-7c950fd26755 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:46:29.792564 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8714752e-3284-49fa-a99f-7c950fd26755 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:29.796116 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-682daed2-59cc-4526-992b-be47fbb0f444 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:29.796755 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682daed2-59cc-4526-992b-be47fbb0f444 service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:29.797041 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682daed2-59cc-4526-992b-be47fbb0f444 service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:29.797319 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682daed2-59cc-4526-992b-be47fbb0f444 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:29.807073 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-682daed2-59cc-4526-992b-be47fbb0f444 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:29.808494 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-682daed2-59cc-4526-992b-be47fbb0f444 service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:29.819266 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1202/2435] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:46:29 2022] GET /identity/v3/auth/tokens => generated 3297 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:46:30.015716 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46587cdb-5638-468e-b09e-1e301000beb2 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:30.025863 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46587cdb-5638-468e-b09e-1e301000beb2 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:30.029721 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0046759c-fa3c-4ce6-b1ed-d831f340fed2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:30.030372 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0046759c-fa3c-4ce6-b1ed-d831f340fed2 service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:30.030743 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0046759c-fa3c-4ce6-b1ed-d831f340fed2 service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:30.030989 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0046759c-fa3c-4ce6-b1ed-d831f340fed2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:30.090281 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0046759c-fa3c-4ce6-b1ed-d831f340fed2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:30.091725 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0046759c-fa3c-4ce6-b1ed-d831f340fed2 service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:30.101075 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1234/2436] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:46:30 2022] GET /identity/v3/auth/tokens => generated 3230 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:46:39.813879 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=49,keystone:INSERT=5 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:46:40.096779 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=97 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:46:41.742823 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-682daed2-59cc-4526-992b-be47fbb0f444 service cinder] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:41.806553 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-682daed2-59cc-4526-992b-be47fbb0f444 service cinder] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:41.811165 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7226d0d-e7f8-454d-bd5e-43e3f2a55db8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:41.812039 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7226d0d-e7f8-454d-bd5e-43e3f2a55db8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:41.812575 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7226d0d-e7f8-454d-bd5e-43e3f2a55db8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:41.812889 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7226d0d-e7f8-454d-bd5e-43e3f2a55db8 admin admin] PATH_INFO: `/v3/users/9bb12c28426f4137b66c7dc75cc38f28` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:41.814086 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7226d0d-e7f8-454d-bd5e-43e3f2a55db8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9bb12c28426f4137b66c7dc75cc38f28)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:41.816088 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7226d0d-e7f8-454d-bd5e-43e3f2a55db8 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:42.011099 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-f7226d0d-e7f8-454d-bd5e-43e3f2a55db8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9bb12c28426f4137b66c7dc75cc38f28'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:42.093251 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-f7226d0d-e7f8-454d-bd5e-43e3f2a55db8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9bb12c28426f4137b66c7dc75cc38f28'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:42.093919 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-f7226d0d-e7f8-454d-bd5e-43e3f2a55db8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9bb12c28426f4137b66c7dc75cc38f28'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:42.101876 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-f7226d0d-e7f8-454d-bd5e-43e3f2a55db8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9bb12c28426f4137b66c7dc75cc38f28'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:42.114287 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-f7226d0d-e7f8-454d-bd5e-43e3f2a55db8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9bb12c28426f4137b66c7dc75cc38f28'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:42.120323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1203/2437] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:46:41 2022] DELETE /identity/v3/users/9bb12c28426f4137b66c7dc75cc38f28 => generated 0 bytes in 378 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:46:42.124994 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ca4fa9-aee8-4768-8301-e0d129c30831 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:42.125498 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ca4fa9-aee8-4768-8301-e0d129c30831 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:42.125498 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ca4fa9-aee8-4768-8301-e0d129c30831 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:42.142626 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-11ca4fa9-aee8-4768-8301-e0d129c30831 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:46:42.144048 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-11ca4fa9-aee8-4768-8301-e0d129c30831 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:46:42.203334 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1235/2438] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:46:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:46:42.213201 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7226d0d-e7f8-454d-bd5e-43e3f2a55db8 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:42.214658 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-f7226d0d-e7f8-454d-bd5e-43e3f2a55db8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:46:42.275309 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7226d0d-e7f8-454d-bd5e-43e3f2a55db8 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:42.279167 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6881b0f9-9b85-4923-899d-e75810800857 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:42.279862 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6881b0f9-9b85-4923-899d-e75810800857 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:42.281738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6881b0f9-9b85-4923-899d-e75810800857 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:42.281738 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6881b0f9-9b85-4923-899d-e75810800857 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:42.290311 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6881b0f9-9b85-4923-899d-e75810800857 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:42.291833 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6881b0f9-9b85-4923-899d-e75810800857 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:42.301986 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1204/2439] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:46:42 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:46:42.745697 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ca4fa9-aee8-4768-8301-e0d129c30831 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:42.747200 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-11ca4fa9-aee8-4768-8301-e0d129c30831 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:46:42.761648 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ca4fa9-aee8-4768-8301-e0d129c30831 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:42.767018 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e111ecf4-0223-42b9-9dc8-182ac9741db3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:42.767890 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e111ecf4-0223-42b9-9dc8-182ac9741db3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:42.768180 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e111ecf4-0223-42b9-9dc8-182ac9741db3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:42.768462 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e111ecf4-0223-42b9-9dc8-182ac9741db3 admin admin] PATH_INFO: `/v3/projects/94676f7e0f51455a8cab7219eb2560e4` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:42.769668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e111ecf4-0223-42b9-9dc8-182ac9741db3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=94676f7e0f51455a8cab7219eb2560e4)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:42.771723 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e111ecf4-0223-42b9-9dc8-182ac9741db3 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:42.891056 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e111ecf4-0223-42b9-9dc8-182ac9741db3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '94676f7e0f51455a8cab7219eb2560e4', 'request_id': 'req-e111ecf4-0223-42b9-9dc8-182ac9741db3'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:42.897365 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e111ecf4-0223-42b9-9dc8-182ac9741db3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '94676f7e0f51455a8cab7219eb2560e4', 'request_id': 'req-e111ecf4-0223-42b9-9dc8-182ac9741db3'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:42.983786 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e111ecf4-0223-42b9-9dc8-182ac9741db3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '94676f7e0f51455a8cab7219eb2560e4', 'request_id': 'req-e111ecf4-0223-42b9-9dc8-182ac9741db3'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:42.997479 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e111ecf4-0223-42b9-9dc8-182ac9741db3 admin admin] The token cache is being invalidate because project 94676f7e0f51455a8cab7219eb2560e4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:46:42.997769 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e111ecf4-0223-42b9-9dc8-182ac9741db3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:42.998467 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e111ecf4-0223-42b9-9dc8-182ac9741db3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:43.000108 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1236/2440] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:46:42 2022] DELETE /identity/v3/projects/94676f7e0f51455a8cab7219eb2560e4 => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:46:43.004377 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6881b0f9-9b85-4923-899d-e75810800857 service neutron] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:43.014498 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6881b0f9-9b85-4923-899d-e75810800857 service neutron] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:43.033233 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:46:43.073420 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef278f19-2dd8-47e6-b138-8284507043e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:43.074306 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef278f19-2dd8-47e6-b138-8284507043e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:43.074630 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef278f19-2dd8-47e6-b138-8284507043e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:43.074917 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef278f19-2dd8-47e6-b138-8284507043e7 admin admin] PATH_INFO: `/v3/users/ca881c33822a4efb9fcb0cb1856572c6` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:43.076141 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef278f19-2dd8-47e6-b138-8284507043e7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ca881c33822a4efb9fcb0cb1856572c6)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:43.078065 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef278f19-2dd8-47e6-b138-8284507043e7 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:43.301474 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-ef278f19-2dd8-47e6-b138-8284507043e7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ca881c33822a4efb9fcb0cb1856572c6'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:43.373425 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-ef278f19-2dd8-47e6-b138-8284507043e7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ca881c33822a4efb9fcb0cb1856572c6'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:43.374220 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-ef278f19-2dd8-47e6-b138-8284507043e7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ca881c33822a4efb9fcb0cb1856572c6'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:43.382545 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-ef278f19-2dd8-47e6-b138-8284507043e7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ca881c33822a4efb9fcb0cb1856572c6'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:43.395343 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-ef278f19-2dd8-47e6-b138-8284507043e7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ca881c33822a4efb9fcb0cb1856572c6'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:43.401327 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1205/2441] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:46:43 2022] DELETE /identity/v3/users/ca881c33822a4efb9fcb0cb1856572c6 => generated 0 bytes in 398 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:46:43.838594 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e111ecf4-0223-42b9-9dc8-182ac9741db3 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:43.840124 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-e111ecf4-0223-42b9-9dc8-182ac9741db3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:46:43.861487 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e111ecf4-0223-42b9-9dc8-182ac9741db3 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:43.921550 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4f84d0f-c4ee-41da-a84c-2ca42cbcf15a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:43.922419 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f84d0f-c4ee-41da-a84c-2ca42cbcf15a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:43.922733 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f84d0f-c4ee-41da-a84c-2ca42cbcf15a admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:43.923014 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f84d0f-c4ee-41da-a84c-2ca42cbcf15a admin admin] PATH_INFO: `/v3/projects/3984ab8f799544ae88219d2cbb87fe95` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:43.924203 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4f84d0f-c4ee-41da-a84c-2ca42cbcf15a admin admin] RBAC: Authorizing `identity:delete_project(project_id=3984ab8f799544ae88219d2cbb87fe95)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:43.926216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4f84d0f-c4ee-41da-a84c-2ca42cbcf15a admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:43.992641 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:46:44.009010 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-c4f84d0f-c4ee-41da-a84c-2ca42cbcf15a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3984ab8f799544ae88219d2cbb87fe95', 'request_id': 'req-c4f84d0f-c4ee-41da-a84c-2ca42cbcf15a'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:44.014444 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-c4f84d0f-c4ee-41da-a84c-2ca42cbcf15a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3984ab8f799544ae88219d2cbb87fe95', 'request_id': 'req-c4f84d0f-c4ee-41da-a84c-2ca42cbcf15a'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:44.105206 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-c4f84d0f-c4ee-41da-a84c-2ca42cbcf15a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3984ab8f799544ae88219d2cbb87fe95', 'request_id': 'req-c4f84d0f-c4ee-41da-a84c-2ca42cbcf15a'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:44.118021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-c4f84d0f-c4ee-41da-a84c-2ca42cbcf15a admin admin] The token cache is being invalidate because project 3984ab8f799544ae88219d2cbb87fe95 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:46:44.118324 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-c4f84d0f-c4ee-41da-a84c-2ca42cbcf15a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:44.119039 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-c4f84d0f-c4ee-41da-a84c-2ca42cbcf15a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:46:44.120686 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1237/2442] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:46:43 2022] DELETE /identity/v3/projects/3984ab8f799544ae88219d2cbb87fe95 => generated 0 bytes in 283 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:46:44.125750 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90ddf54-9614-4d56-a967-197b77d61786 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:44.126241 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90ddf54-9614-4d56-a967-197b77d61786 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:44.126355 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90ddf54-9614-4d56-a967-197b77d61786 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:44.145462 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-f90ddf54-9614-4d56-a967-197b77d61786 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:46:44.146702 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-f90ddf54-9614-4d56-a967-197b77d61786 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:46:44.205854 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1206/2443] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:46:44 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:46:44.213937 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc1064a-403e-4e01-ad76-d9524ed6d39e None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:44.214228 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc1064a-403e-4e01-ad76-d9524ed6d39e None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:44.214541 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc1064a-403e-4e01-ad76-d9524ed6d39e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:44.232251 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-cbc1064a-403e-4e01-ad76-d9524ed6d39e None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:46:44.233440 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-cbc1064a-403e-4e01-ad76-d9524ed6d39e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:46:44.241868 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1238/2444] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:46:44 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:46:44.246124 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90ddf54-9614-4d56-a967-197b77d61786 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:44.256613 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90ddf54-9614-4d56-a967-197b77d61786 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:44.261270 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc45333-dac1-4fc7-b95b-cbf9b7483b7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:44.261914 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc45333-dac1-4fc7-b95b-cbf9b7483b7c admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:44.262206 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc45333-dac1-4fc7-b95b-cbf9b7483b7c admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:44.262523 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc45333-dac1-4fc7-b95b-cbf9b7483b7c admin admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:44.263255 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc45333-dac1-4fc7-b95b-cbf9b7483b7c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:46:44.263695 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc45333-dac1-4fc7-b95b-cbf9b7483b7c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:44.265172 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc45333-dac1-4fc7-b95b-cbf9b7483b7c admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:44.277317 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1207/2445] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:46:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:46:44.281859 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc1064a-403e-4e01-ad76-d9524ed6d39e None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:44.291893 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc1064a-403e-4e01-ad76-d9524ed6d39e None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:44.296755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb36edaa-ed32-490b-9765-fb2b93ede20a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:44.297403 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb36edaa-ed32-490b-9765-fb2b93ede20a admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:44.297686 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb36edaa-ed32-490b-9765-fb2b93ede20a admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:44.297963 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb36edaa-ed32-490b-9765-fb2b93ede20a admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:44.298681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb36edaa-ed32-490b-9765-fb2b93ede20a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:44.300570 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb36edaa-ed32-490b-9765-fb2b93ede20a admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:44.478184 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1239/2446] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:46:44 2022] POST /identity/v3/projects => generated 377 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:46:44.482373 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc45333-dac1-4fc7-b95b-cbf9b7483b7c admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:44.493359 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc45333-dac1-4fc7-b95b-cbf9b7483b7c admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:44.519337 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:46:44.549585 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c05ae1-f756-4277-a8a6-4c9f5a5c03c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:44.550253 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c05ae1-f756-4277-a8a6-4c9f5a5c03c6 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:44.550576 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c05ae1-f756-4277-a8a6-4c9f5a5c03c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:44.550855 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c05ae1-f756-4277-a8a6-4c9f5a5c03c6 admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:44.552271 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c05ae1-f756-4277-a8a6-4c9f5a5c03c6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:44.554175 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c05ae1-f756-4277-a8a6-4c9f5a5c03c6 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:44.686237 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1208/2447] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:46:44 2022] POST /identity/v3/users => generated 289 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:46:44.687619 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb36edaa-ed32-490b-9765-fb2b93ede20a admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:44.698481 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb36edaa-ed32-490b-9765-fb2b93ede20a admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:44.703020 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b37594-c953-4387-be7d-4e86eeed5407 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:44.703676 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b37594-c953-4387-be7d-4e86eeed5407 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:44.703964 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b37594-c953-4387-be7d-4e86eeed5407 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:44.704242 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b37594-c953-4387-be7d-4e86eeed5407 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:44.704976 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b37594-c953-4387-be7d-4e86eeed5407 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:44.706596 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b37594-c953-4387-be7d-4e86eeed5407 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:44.717149 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1240/2448] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:46:44 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 12 17:46:44.721360 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c05ae1-f756-4277-a8a6-4c9f5a5c03c6 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:44.732914 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c05ae1-f756-4277-a8a6-4c9f5a5c03c6 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:44.736980 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ceea7a-0bd3-4d59-a289-87001feb1eec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:44.737649 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ceea7a-0bd3-4d59-a289-87001feb1eec admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:44.737940 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ceea7a-0bd3-4d59-a289-87001feb1eec admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:44.738223 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ceea7a-0bd3-4d59-a289-87001feb1eec admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:44.739787 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3ceea7a-0bd3-4d59-a289-87001feb1eec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:44.741380 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3ceea7a-0bd3-4d59-a289-87001feb1eec admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:44.752441 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1209/2449] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:46:44 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:46:44.756217 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b37594-c953-4387-be7d-4e86eeed5407 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:44.766307 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b37594-c953-4387-be7d-4e86eeed5407 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:44.771655 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c809ef-d8e8-4a3c-b15d-9d48cc5517fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:44.772527 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c809ef-d8e8-4a3c-b15d-9d48cc5517fe admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:44.772818 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c809ef-d8e8-4a3c-b15d-9d48cc5517fe admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:44.773098 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c809ef-d8e8-4a3c-b15d-9d48cc5517fe admin admin] PATH_INFO: `/v3/projects/a5f461a4f9ca421ba6e503aa160570f2/users/035be849cdf64c1590c37ec0e75e892d/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:44.789761 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9c809ef-d8e8-4a3c-b15d-9d48cc5517fe admin admin] RBAC: Authorizing `identity:create_grant(project_id=a5f461a4f9ca421ba6e503aa160570f2, user_id=035be849cdf64c1590c37ec0e75e892d, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:44.792429 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9c809ef-d8e8-4a3c-b15d-9d48cc5517fe admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:44.847695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1241/2450] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:46:44 2022] PUT /identity/v3/projects/a5f461a4f9ca421ba6e503aa160570f2/users/035be849cdf64c1590c37ec0e75e892d/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 92 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:46:44.857999 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23234423-e857-4853-b544-338d2ff1f8da None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:44.860644 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23234423-e857-4853-b544-338d2ff1f8da None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:44.860644 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23234423-e857-4853-b544-338d2ff1f8da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:44.886502 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-23234423-e857-4853-b544-338d2ff1f8da None None] MFA Rules not processed for user `035be849cdf64c1590c37ec0e75e892d`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:46:44.887722 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-23234423-e857-4853-b544-338d2ff1f8da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:46:44.952478 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1210/2451] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:46:44 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:46:44.956979 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c809ef-d8e8-4a3c-b15d-9d48cc5517fe admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:45.018492 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c809ef-d8e8-4a3c-b15d-9d48cc5517fe admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:45.023145 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11faa239-3888-4192-87ba-7ff7efdb9e9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:45.023798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11faa239-3888-4192-87ba-7ff7efdb9e9e admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:45.024095 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11faa239-3888-4192-87ba-7ff7efdb9e9e admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:45.024401 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11faa239-3888-4192-87ba-7ff7efdb9e9e admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:45.025096 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11faa239-3888-4192-87ba-7ff7efdb9e9e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:45.027045 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11faa239-3888-4192-87ba-7ff7efdb9e9e admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:45.134567 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1242/2452] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:46:44 2022] POST /identity/v3/projects => generated 375 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:46:45.138728 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23234423-e857-4853-b544-338d2ff1f8da None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:45.150008 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23234423-e857-4853-b544-338d2ff1f8da None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:45.209117 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc22171-41db-4807-8775-6de0897a48f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:45.209819 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc22171-41db-4807-8775-6de0897a48f5 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:45.210108 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc22171-41db-4807-8775-6de0897a48f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:45.210388 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc22171-41db-4807-8775-6de0897a48f5 admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:45.211109 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc22171-41db-4807-8775-6de0897a48f5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:45.213028 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc22171-41db-4807-8775-6de0897a48f5 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:45.334238 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1211/2453] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:46:45 2022] POST /identity/v3/users => generated 294 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:46:45.343147 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11faa239-3888-4192-87ba-7ff7efdb9e9e admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:45.354168 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11faa239-3888-4192-87ba-7ff7efdb9e9e admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:45.358820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee30569c-8bfd-4c82-8c52-24ebad547ff8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:45.359491 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee30569c-8bfd-4c82-8c52-24ebad547ff8 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:45.359760 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee30569c-8bfd-4c82-8c52-24ebad547ff8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:45.360068 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee30569c-8bfd-4c82-8c52-24ebad547ff8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:45.360837 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee30569c-8bfd-4c82-8c52-24ebad547ff8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:45.362453 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee30569c-8bfd-4c82-8c52-24ebad547ff8 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:45.373972 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1243/2454] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:46:45 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:46:45.378282 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc22171-41db-4807-8775-6de0897a48f5 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:45.388411 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc22171-41db-4807-8775-6de0897a48f5 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:45.393089 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62369446-4ff5-4d54-8d70-c125bf541c68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:45.393968 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62369446-4ff5-4d54-8d70-c125bf541c68 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:45.394266 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62369446-4ff5-4d54-8d70-c125bf541c68 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:45.394584 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62369446-4ff5-4d54-8d70-c125bf541c68 admin admin] PATH_INFO: `/v3/projects/86a5feac6dcf4533a9dbefa617eaef06/users/3a569e35fbea448fa651a885a390b594/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:45.409688 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62369446-4ff5-4d54-8d70-c125bf541c68 admin admin] RBAC: Authorizing `identity:create_grant(project_id=86a5feac6dcf4533a9dbefa617eaef06, user_id=3a569e35fbea448fa651a885a390b594, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:45.413109 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62369446-4ff5-4d54-8d70-c125bf541c68 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:45.522664 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1212/2455] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:46:45 2022] PUT /identity/v3/projects/86a5feac6dcf4533a9dbefa617eaef06/users/3a569e35fbea448fa651a885a390b594/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:46:45.526680 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee30569c-8bfd-4c82-8c52-24ebad547ff8 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:45.537919 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee30569c-8bfd-4c82-8c52-24ebad547ff8 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:45.557602 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:46:45.597011 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe715f4-aec1-4cb9-9dcd-999a42b6f7c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:45.597643 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe715f4-aec1-4cb9-9dcd-999a42b6f7c7 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:45.598612 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe715f4-aec1-4cb9-9dcd-999a42b6f7c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:45.598999 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe715f4-aec1-4cb9-9dcd-999a42b6f7c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:45.599872 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe715f4-aec1-4cb9-9dcd-999a42b6f7c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:45.601497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe715f4-aec1-4cb9-9dcd-999a42b6f7c7 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:45.612485 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1244/2456] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:46:45 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:46:45.616706 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62369446-4ff5-4d54-8d70-c125bf541c68 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:45.622419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:46:45.628072 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62369446-4ff5-4d54-8d70-c125bf541c68 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:45.633851 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a919917a-407e-4418-9072-f47c15bf8719 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:45.634155 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a919917a-407e-4418-9072-f47c15bf8719 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:45.634470 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a919917a-407e-4418-9072-f47c15bf8719 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:45.634759 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a919917a-407e-4418-9072-f47c15bf8719 admin admin] PATH_INFO: `/v3/projects/86a5feac6dcf4533a9dbefa617eaef06/users/3a569e35fbea448fa651a885a390b594/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:45.636794 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a919917a-407e-4418-9072-f47c15bf8719 admin admin] RBAC: Authorizing `identity:create_grant(project_id=86a5feac6dcf4533a9dbefa617eaef06, user_id=3a569e35fbea448fa651a885a390b594, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:45.639474 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a919917a-407e-4418-9072-f47c15bf8719 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:45.651420 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1213/2457] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:46:45 2022] PUT /identity/v3/projects/86a5feac6dcf4533a9dbefa617eaef06/users/3a569e35fbea448fa651a885a390b594/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:46:45.663791 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd2faaa-ed2d-4e80-99bb-587df9e45bfb None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:45.664125 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd2faaa-ed2d-4e80-99bb-587df9e45bfb None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:45.664420 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd2faaa-ed2d-4e80-99bb-587df9e45bfb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:45.691497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-0fd2faaa-ed2d-4e80-99bb-587df9e45bfb None None] MFA Rules not processed for user `3a569e35fbea448fa651a885a390b594`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:46:45.694299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-0fd2faaa-ed2d-4e80-99bb-587df9e45bfb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:46:45.763510 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1245/2458] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:46:45 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:46:45.775748 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a919917a-407e-4418-9072-f47c15bf8719 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:45.777563 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-a919917a-407e-4418-9072-f47c15bf8719 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:46:45.836853 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a919917a-407e-4418-9072-f47c15bf8719 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:45.840456 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbba6b5e-7f38-4491-ae76-d7ada59f20a1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:45.841096 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbba6b5e-7f38-4491-ae76-d7ada59f20a1 service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:45.841380 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbba6b5e-7f38-4491-ae76-d7ada59f20a1 service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:45.841657 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbba6b5e-7f38-4491-ae76-d7ada59f20a1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:45.897953 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbba6b5e-7f38-4491-ae76-d7ada59f20a1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:45.899402 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbba6b5e-7f38-4491-ae76-d7ada59f20a1 service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:45.908171 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1214/2459] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:46:45 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:46:52.122389 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd2faaa-ed2d-4e80-99bb-587df9e45bfb None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:46:52.132723 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd2faaa-ed2d-4e80-99bb-587df9e45bfb None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:46:52.136339 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f66f16-0980-4285-844e-a20af5bdfdcf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:46:52.136981 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f66f16-0980-4285-844e-a20af5bdfdcf service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:46:52.137269 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f66f16-0980-4285-844e-a20af5bdfdcf service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:46:52.137550 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f66f16-0980-4285-844e-a20af5bdfdcf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:46:52.147503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8f66f16-0980-4285-844e-a20af5bdfdcf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:46:52.148935 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8f66f16-0980-4285-844e-a20af5bdfdcf service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:46:52.158407 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1246/2460] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:46:52 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:46:55.904303 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=52,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:02.154176 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=58 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:03.101871 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbba6b5e-7f38-4491-ae76-d7ada59f20a1 service cinder] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:03.103387 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-bbba6b5e-7f38-4491-ae76-d7ada59f20a1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:03.208666 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbba6b5e-7f38-4491-ae76-d7ada59f20a1 service cinder] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:03.214084 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f00239-2c9d-453e-9c5e-860e8c258160 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:03.214801 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f00239-2c9d-453e-9c5e-860e8c258160 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:03.215114 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f00239-2c9d-453e-9c5e-860e8c258160 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:03.215434 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f00239-2c9d-453e-9c5e-860e8c258160 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:03.217577 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-f8f00239-2c9d-453e-9c5e-860e8c258160 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:03.329313 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f00239-2c9d-453e-9c5e-860e8c258160 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:03.330747 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f00239-2c9d-453e-9c5e-860e8c258160 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:03.352558 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1215/2461] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:47:03 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 251 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:47:07.020941 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f66f16-0980-4285-844e-a20af5bdfdcf service cinder] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:07.022360 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-c8f66f16-0980-4285-844e-a20af5bdfdcf service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:07.059901 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f66f16-0980-4285-844e-a20af5bdfdcf service cinder] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:07.064447 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-640465d0-58f6-451e-9586-0a048aa9382b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:07.066081 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640465d0-58f6-451e-9586-0a048aa9382b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:07.066081 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640465d0-58f6-451e-9586-0a048aa9382b admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:07.066081 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640465d0-58f6-451e-9586-0a048aa9382b admin admin] PATH_INFO: `/v3/users/9feeaf257bb441b7a14b57f1f493d08f` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:07.067826 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-640465d0-58f6-451e-9586-0a048aa9382b admin admin] RBAC: Authorizing `identity:delete_user(user_id=9feeaf257bb441b7a14b57f1f493d08f)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:07.070024 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-640465d0-58f6-451e-9586-0a048aa9382b admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:07.210614 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-640465d0-58f6-451e-9586-0a048aa9382b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9feeaf257bb441b7a14b57f1f493d08f'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:07.211275 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-640465d0-58f6-451e-9586-0a048aa9382b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9feeaf257bb441b7a14b57f1f493d08f'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:07.211887 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-640465d0-58f6-451e-9586-0a048aa9382b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9feeaf257bb441b7a14b57f1f493d08f'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:07.219047 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-640465d0-58f6-451e-9586-0a048aa9382b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9feeaf257bb441b7a14b57f1f493d08f'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:07.230895 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-640465d0-58f6-451e-9586-0a048aa9382b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9feeaf257bb441b7a14b57f1f493d08f'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:07.326560 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1247/2462] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:47:07 2022] DELETE /identity/v3/users/9feeaf257bb441b7a14b57f1f493d08f => generated 0 bytes in 307 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:07.632916 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f00239-2c9d-453e-9c5e-860e8c258160 service neutron] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:07.643550 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f00239-2c9d-453e-9c5e-860e8c258160 service neutron] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:07.676657 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:07.705379 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50dd871d-5602-4e55-b9ae-c5aa2483b7e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:07.706265 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50dd871d-5602-4e55-b9ae-c5aa2483b7e2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:07.706690 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50dd871d-5602-4e55-b9ae-c5aa2483b7e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:07.707002 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50dd871d-5602-4e55-b9ae-c5aa2483b7e2 admin admin] PATH_INFO: `/v3/projects/49227e443aa94a438bf7954aa0398cb9` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:07.708237 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50dd871d-5602-4e55-b9ae-c5aa2483b7e2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=49227e443aa94a438bf7954aa0398cb9)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:07.710303 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50dd871d-5602-4e55-b9ae-c5aa2483b7e2 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:07.820621 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-50dd871d-5602-4e55-b9ae-c5aa2483b7e2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '49227e443aa94a438bf7954aa0398cb9', 'request_id': 'req-50dd871d-5602-4e55-b9ae-c5aa2483b7e2'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:07.827200 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-50dd871d-5602-4e55-b9ae-c5aa2483b7e2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '49227e443aa94a438bf7954aa0398cb9', 'request_id': 'req-50dd871d-5602-4e55-b9ae-c5aa2483b7e2'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:07.835011 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-50dd871d-5602-4e55-b9ae-c5aa2483b7e2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '49227e443aa94a438bf7954aa0398cb9', 'request_id': 'req-50dd871d-5602-4e55-b9ae-c5aa2483b7e2'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:07.904730 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-50dd871d-5602-4e55-b9ae-c5aa2483b7e2 admin admin] The token cache is being invalidate because project 49227e443aa94a438bf7954aa0398cb9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:47:07.905172 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-50dd871d-5602-4e55-b9ae-c5aa2483b7e2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:07.905843 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-50dd871d-5602-4e55-b9ae-c5aa2483b7e2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:07.907547 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1216/2463] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:47:07 2022] DELETE /identity/v3/projects/49227e443aa94a438bf7954aa0398cb9 => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:47:07.913386 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9175a209-0f1c-4498-93d0-85ba117b0381 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:07.913682 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9175a209-0f1c-4498-93d0-85ba117b0381 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:07.913966 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9175a209-0f1c-4498-93d0-85ba117b0381 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:07.976572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-9175a209-0f1c-4498-93d0-85ba117b0381 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:07.977863 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-9175a209-0f1c-4498-93d0-85ba117b0381 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:08.043295 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1248/2464] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:47:08.051674 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0033b474-af31-4154-a19a-c7a2ad8f2f92 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:08.052146 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0033b474-af31-4154-a19a-c7a2ad8f2f92 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:08.052639 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0033b474-af31-4154-a19a-c7a2ad8f2f92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:08.074894 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-0033b474-af31-4154-a19a-c7a2ad8f2f92 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:08.076152 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-0033b474-af31-4154-a19a-c7a2ad8f2f92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:08.088917 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1217/2465] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:47:08.090140 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9175a209-0f1c-4498-93d0-85ba117b0381 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:08.101642 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9175a209-0f1c-4498-93d0-85ba117b0381 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:08.107093 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37fa3959-7c3e-4fd4-8c65-3a15e9fa5cd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:08.107884 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fa3959-7c3e-4fd4-8c65-3a15e9fa5cd7 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:08.108270 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fa3959-7c3e-4fd4-8c65-3a15e9fa5cd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:08.108790 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fa3959-7c3e-4fd4-8c65-3a15e9fa5cd7 admin admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:08.109723 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37fa3959-7c3e-4fd4-8c65-3a15e9fa5cd7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:47:08.110134 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37fa3959-7c3e-4fd4-8c65-3a15e9fa5cd7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:08.112408 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37fa3959-7c3e-4fd4-8c65-3a15e9fa5cd7 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:08.125899 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1249/2466] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:47:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:47:08.130805 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0033b474-af31-4154-a19a-c7a2ad8f2f92 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:08.140459 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0033b474-af31-4154-a19a-c7a2ad8f2f92 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:08.145759 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2e4e58-4dbf-4c27-a445-032286692dc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:08.146462 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2e4e58-4dbf-4c27-a445-032286692dc2 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:08.146777 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2e4e58-4dbf-4c27-a445-032286692dc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:08.147061 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2e4e58-4dbf-4c27-a445-032286692dc2 admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:08.147764 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b2e4e58-4dbf-4c27-a445-032286692dc2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:08.149683 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b2e4e58-4dbf-4c27-a445-032286692dc2 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:08.216277 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1218/2467] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:47:08 2022] POST /identity/v3/projects => generated 383 bytes in 86 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:47:08.220890 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37fa3959-7c3e-4fd4-8c65-3a15e9fa5cd7 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:08.228920 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:08.232902 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37fa3959-7c3e-4fd4-8c65-3a15e9fa5cd7 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:08.292908 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1761cd6b-5856-457c-8026-49340f46c860 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:08.293686 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1761cd6b-5856-457c-8026-49340f46c860 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:08.293983 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1761cd6b-5856-457c-8026-49340f46c860 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:08.294295 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1761cd6b-5856-457c-8026-49340f46c860 admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:08.295044 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1761cd6b-5856-457c-8026-49340f46c860 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:08.296973 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1761cd6b-5856-457c-8026-49340f46c860 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:08.374245 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1250/2468] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:47:08 2022] POST /identity/v3/users => generated 292 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:47:08.378540 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2e4e58-4dbf-4c27-a445-032286692dc2 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:08.389712 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2e4e58-4dbf-4c27-a445-032286692dc2 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:08.395147 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae4e6f88-ca98-4b18-8d70-ea2a79916907 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:08.395828 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4e6f88-ca98-4b18-8d70-ea2a79916907 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:08.396231 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4e6f88-ca98-4b18-8d70-ea2a79916907 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:08.396522 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4e6f88-ca98-4b18-8d70-ea2a79916907 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:08.397260 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae4e6f88-ca98-4b18-8d70-ea2a79916907 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:08.398903 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae4e6f88-ca98-4b18-8d70-ea2a79916907 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:08.409625 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1219/2469] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:47:08 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:47:08.413979 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1761cd6b-5856-457c-8026-49340f46c860 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:08.424106 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1761cd6b-5856-457c-8026-49340f46c860 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:08.429462 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02cabdd9-dd71-4f63-b28d-4a125fd92bad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:08.430122 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cabdd9-dd71-4f63-b28d-4a125fd92bad admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:08.430415 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cabdd9-dd71-4f63-b28d-4a125fd92bad admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:08.430724 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cabdd9-dd71-4f63-b28d-4a125fd92bad admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:08.431469 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02cabdd9-dd71-4f63-b28d-4a125fd92bad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:08.433075 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02cabdd9-dd71-4f63-b28d-4a125fd92bad admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:08.444818 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1251/2470] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:47:08 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:47:08.449200 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae4e6f88-ca98-4b18-8d70-ea2a79916907 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:08.460135 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae4e6f88-ca98-4b18-8d70-ea2a79916907 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:08.464787 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-647b502a-8af3-4083-8122-f8d15a1b0d0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:08.465685 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647b502a-8af3-4083-8122-f8d15a1b0d0e admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:08.465975 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647b502a-8af3-4083-8122-f8d15a1b0d0e admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:08.466259 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-647b502a-8af3-4083-8122-f8d15a1b0d0e admin admin] PATH_INFO: `/v3/projects/9f8d28bb1201459882314009cad942b3/users/b1caccaedb6d41d1bbc4e2180c73efff/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:08.481582 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-647b502a-8af3-4083-8122-f8d15a1b0d0e admin admin] RBAC: Authorizing `identity:create_grant(project_id=9f8d28bb1201459882314009cad942b3, user_id=b1caccaedb6d41d1bbc4e2180c73efff, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:08.484293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-647b502a-8af3-4083-8122-f8d15a1b0d0e admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:08.539103 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1220/2471] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:47:08 2022] PUT /identity/v3/projects/9f8d28bb1201459882314009cad942b3/users/b1caccaedb6d41d1bbc4e2180c73efff/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:08.550388 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdcd1f5-d9fb-410b-bbf3-29d4cdb5be4c None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:08.550770 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdcd1f5-d9fb-410b-bbf3-29d4cdb5be4c None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:08.551054 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fdcd1f5-d9fb-410b-bbf3-29d4cdb5be4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:08.579693 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-2fdcd1f5-d9fb-410b-bbf3-29d4cdb5be4c None None] MFA Rules not processed for user `b1caccaedb6d41d1bbc4e2180c73efff`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:08.580927 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-2fdcd1f5-d9fb-410b-bbf3-29d4cdb5be4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:08.654404 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1252/2472] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:08 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:47:08.665906 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-647b502a-8af3-4083-8122-f8d15a1b0d0e admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:08.667753 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-647b502a-8af3-4083-8122-f8d15a1b0d0e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:08.699103 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fdcd1f5-d9fb-410b-bbf3-29d4cdb5be4c None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:08.702992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-2fdcd1f5-d9fb-410b-bbf3-29d4cdb5be4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:08.736328 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:08.778699 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-647b502a-8af3-4083-8122-f8d15a1b0d0e admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:08.783247 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:08.784612 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3140c2-c4dc-4816-b388-c21f705a677d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:08.785363 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3140c2-c4dc-4816-b388-c21f705a677d service nova] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:08.785599 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3140c2-c4dc-4816-b388-c21f705a677d service nova] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:08.785881 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a3140c2-c4dc-4816-b388-c21f705a677d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:08.794549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a3140c2-c4dc-4816-b388-c21f705a677d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:08.796165 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a3140c2-c4dc-4816-b388-c21f705a677d service nova] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:08.797278 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fdcd1f5-d9fb-410b-bbf3-29d4cdb5be4c None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:08.803726 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920453c0-2f25-4f0b-b0ee-b463c21741d2 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f623be28c83247c78d3ea6873c347fe0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:08.804537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920453c0-2f25-4f0b-b0ee-b463c21741d2 service placement] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:08.804918 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920453c0-2f25-4f0b-b0ee-b463c21741d2 service placement] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:08.805311 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920453c0-2f25-4f0b-b0ee-b463c21741d2 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:08.806556 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1221/2473] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:47:08 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:47:08.811233 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-920453c0-2f25-4f0b-b0ee-b463c21741d2 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:08.825127 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-920453c0-2f25-4f0b-b0ee-b463c21741d2 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:08.826816 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-920453c0-2f25-4f0b-b0ee-b463c21741d2 service placement] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:08.838679 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1253/2474] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:47:08 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:47:17.030004 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3140c2-c4dc-4816-b388-c21f705a677d service nova] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:17.031522 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-4a3140c2-c4dc-4816-b388-c21f705a677d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:17.090301 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a3140c2-c4dc-4816-b388-c21f705a677d service nova] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:17.094079 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64dad83a-cb01-4d65-9ffa-2e8b9c32a34a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:17.094779 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64dad83a-cb01-4d65-9ffa-2e8b9c32a34a service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:17.095058 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64dad83a-cb01-4d65-9ffa-2e8b9c32a34a service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:17.095346 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64dad83a-cb01-4d65-9ffa-2e8b9c32a34a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:17.097118 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-64dad83a-cb01-4d65-9ffa-2e8b9c32a34a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:17.157836 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64dad83a-cb01-4d65-9ffa-2e8b9c32a34a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:17.159307 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64dad83a-cb01-4d65-9ffa-2e8b9c32a34a service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:17.168452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1222/2475] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:47:17 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:47:18.914764 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=8 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:22.609719 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920453c0-2f25-4f0b-b0ee-b463c21741d2 service placement] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:22.674483 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920453c0-2f25-4f0b-b0ee-b463c21741d2 service placement] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:22.679390 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6e859a-fcb2-4cc7-965a-d34b036a51df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:22.681131 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6e859a-fcb2-4cc7-965a-d34b036a51df admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:22.681525 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6e859a-fcb2-4cc7-965a-d34b036a51df admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:22.681900 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6e859a-fcb2-4cc7-965a-d34b036a51df admin admin] PATH_INFO: `/v3/users/b1caccaedb6d41d1bbc4e2180c73efff` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:22.683304 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a6e859a-fcb2-4cc7-965a-d34b036a51df admin admin] RBAC: Authorizing `identity:delete_user(user_id=b1caccaedb6d41d1bbc4e2180c73efff)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:22.685343 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a6e859a-fcb2-4cc7-965a-d34b036a51df admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:22.872236 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-9a6e859a-fcb2-4cc7-965a-d34b036a51df admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b1caccaedb6d41d1bbc4e2180c73efff'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:22.872920 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-9a6e859a-fcb2-4cc7-965a-d34b036a51df admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b1caccaedb6d41d1bbc4e2180c73efff'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:22.873523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-9a6e859a-fcb2-4cc7-965a-d34b036a51df admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b1caccaedb6d41d1bbc4e2180c73efff'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:22.880628 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-9a6e859a-fcb2-4cc7-965a-d34b036a51df admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b1caccaedb6d41d1bbc4e2180c73efff'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:22.893189 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-9a6e859a-fcb2-4cc7-965a-d34b036a51df admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b1caccaedb6d41d1bbc4e2180c73efff'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:22.972380 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1254/2476] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:47:22 2022] DELETE /identity/v3/users/b1caccaedb6d41d1bbc4e2180c73efff => generated 0 bytes in 366 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:22.977601 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798cb0e8-f764-47a2-87bb-7582f84376de None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:22.978083 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798cb0e8-f764-47a2-87bb-7582f84376de None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:22.978217 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798cb0e8-f764-47a2-87bb-7582f84376de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:22.995673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-798cb0e8-f764-47a2-87bb-7582f84376de None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:22.996881 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-798cb0e8-f764-47a2-87bb-7582f84376de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:23.056628 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1223/2477] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:47:23.066477 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6e859a-fcb2-4cc7-965a-d34b036a51df admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:23.068031 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-9a6e859a-fcb2-4cc7-965a-d34b036a51df admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:23.147797 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6e859a-fcb2-4cc7-965a-d34b036a51df admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:23.152556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1cdced1-1db7-48f8-9929-af8250f73f2a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:23.153340 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cdced1-1db7-48f8-9929-af8250f73f2a service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:23.153663 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cdced1-1db7-48f8-9929-af8250f73f2a service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:23.153947 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cdced1-1db7-48f8-9929-af8250f73f2a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:23.163639 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1cdced1-1db7-48f8-9929-af8250f73f2a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:23.165997 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1cdced1-1db7-48f8-9929-af8250f73f2a service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:23.176913 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1255/2478] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:47:23 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:47:23.577016 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798cb0e8-f764-47a2-87bb-7582f84376de None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:23.578389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-798cb0e8-f764-47a2-87bb-7582f84376de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:23.591909 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798cb0e8-f764-47a2-87bb-7582f84376de None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:23.600961 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c941a9ce-f0a3-4ad0-8ad9-af9ebff5d244 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:23.600961 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c941a9ce-f0a3-4ad0-8ad9-af9ebff5d244 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:23.600961 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c941a9ce-f0a3-4ad0-8ad9-af9ebff5d244 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:23.600961 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c941a9ce-f0a3-4ad0-8ad9-af9ebff5d244 admin admin] PATH_INFO: `/v3/projects/9f8d28bb1201459882314009cad942b3` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:23.601589 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c941a9ce-f0a3-4ad0-8ad9-af9ebff5d244 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9f8d28bb1201459882314009cad942b3)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:23.601589 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c941a9ce-f0a3-4ad0-8ad9-af9ebff5d244 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:23.611768 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:23.638043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1cdced1-1db7-48f8-9929-af8250f73f2a service neutron] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:23.639476 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-d1cdced1-1db7-48f8-9929-af8250f73f2a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:23.647249 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:23.652583 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1cdced1-1db7-48f8-9929-af8250f73f2a service neutron] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:23.657163 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24cc2cfa-5771-43fd-87d2-6dfba4a7817f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:23.658013 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cc2cfa-5771-43fd-87d2-6dfba4a7817f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:23.658297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cc2cfa-5771-43fd-87d2-6dfba4a7817f admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:23.658683 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cc2cfa-5771-43fd-87d2-6dfba4a7817f admin admin] PATH_INFO: `/v3/users/035be849cdf64c1590c37ec0e75e892d` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:23.659882 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24cc2cfa-5771-43fd-87d2-6dfba4a7817f admin admin] RBAC: Authorizing `identity:delete_user(user_id=035be849cdf64c1590c37ec0e75e892d)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:23.661770 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24cc2cfa-5771-43fd-87d2-6dfba4a7817f admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:23.716067 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-c941a9ce-f0a3-4ad0-8ad9-af9ebff5d244 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9f8d28bb1201459882314009cad942b3', 'request_id': 'req-c941a9ce-f0a3-4ad0-8ad9-af9ebff5d244'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:23.722297 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-c941a9ce-f0a3-4ad0-8ad9-af9ebff5d244 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9f8d28bb1201459882314009cad942b3', 'request_id': 'req-c941a9ce-f0a3-4ad0-8ad9-af9ebff5d244'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:23.727996 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-c941a9ce-f0a3-4ad0-8ad9-af9ebff5d244 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9f8d28bb1201459882314009cad942b3', 'request_id': 'req-c941a9ce-f0a3-4ad0-8ad9-af9ebff5d244'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:23.772644 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-c941a9ce-f0a3-4ad0-8ad9-af9ebff5d244 admin admin] The token cache is being invalidate because project 9f8d28bb1201459882314009cad942b3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:47:23.772942 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-c941a9ce-f0a3-4ad0-8ad9-af9ebff5d244 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:23.773888 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-c941a9ce-f0a3-4ad0-8ad9-af9ebff5d244 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:23.775543 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1224/2479] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:47:23 2022] DELETE /identity/v3/projects/9f8d28bb1201459882314009cad942b3 => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:23.780184 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bffa43a-ebc9-45b3-8a16-c68d1c6e22db None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:23.780479 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bffa43a-ebc9-45b3-8a16-c68d1c6e22db None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:23.780769 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bffa43a-ebc9-45b3-8a16-c68d1c6e22db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:23.799198 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-2bffa43a-ebc9-45b3-8a16-c68d1c6e22db None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:23.800360 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-2bffa43a-ebc9-45b3-8a16-c68d1c6e22db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:23.833577 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-24cc2cfa-5771-43fd-87d2-6dfba4a7817f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '035be849cdf64c1590c37ec0e75e892d'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:23.834318 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-24cc2cfa-5771-43fd-87d2-6dfba4a7817f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '035be849cdf64c1590c37ec0e75e892d'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:23.835003 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-24cc2cfa-5771-43fd-87d2-6dfba4a7817f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '035be849cdf64c1590c37ec0e75e892d'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:23.842164 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-24cc2cfa-5771-43fd-87d2-6dfba4a7817f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '035be849cdf64c1590c37ec0e75e892d'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:23.862792 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1225/2480] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:47:23.871075 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43efe421-bc61-400c-9244-8c4144c163a1 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:23.871275 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-24cc2cfa-5771-43fd-87d2-6dfba4a7817f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '035be849cdf64c1590c37ec0e75e892d'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:23.871370 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43efe421-bc61-400c-9244-8c4144c163a1 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:23.871644 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43efe421-bc61-400c-9244-8c4144c163a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:23.891353 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-43efe421-bc61-400c-9244-8c4144c163a1 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:23.892620 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-43efe421-bc61-400c-9244-8c4144c163a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:23.900727 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1226/2481] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:47:23.904796 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43efe421-bc61-400c-9244-8c4144c163a1 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:23.914345 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43efe421-bc61-400c-9244-8c4144c163a1 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:23.919113 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037a2db9-4bef-4956-ab59-a60295289da2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:23.921827 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037a2db9-4bef-4956-ab59-a60295289da2 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:23.921827 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037a2db9-4bef-4956-ab59-a60295289da2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:23.921827 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037a2db9-4bef-4956-ab59-a60295289da2 admin admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:23.921827 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-037a2db9-4bef-4956-ab59-a60295289da2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:47:23.921827 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-037a2db9-4bef-4956-ab59-a60295289da2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:23.922929 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-037a2db9-4bef-4956-ab59-a60295289da2 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:23.938869 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1227/2482] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:47:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:47:23.940094 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037a2db9-4bef-4956-ab59-a60295289da2 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:23.949750 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1256/2483] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:47:23 2022] DELETE /identity/v3/users/035be849cdf64c1590c37ec0e75e892d => generated 0 bytes in 312 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:47:23.951536 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037a2db9-4bef-4956-ab59-a60295289da2 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:23.954228 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c63c56-dc98-4f09-a0d6-a6368a956a90 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:23.954561 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c63c56-dc98-4f09-a0d6-a6368a956a90 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:23.954848 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c63c56-dc98-4f09-a0d6-a6368a956a90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:23.974152 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-14c63c56-dc98-4f09-a0d6-a6368a956a90 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:23.975789 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-14c63c56-dc98-4f09-a0d6-a6368a956a90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:24.018146 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af849359-d958-4a7e-a340-bf9735e9edcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:24.018859 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af849359-d958-4a7e-a340-bf9735e9edcb admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:24.019201 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af849359-d958-4a7e-a340-bf9735e9edcb admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:24.019488 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af849359-d958-4a7e-a340-bf9735e9edcb admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:24.020199 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af849359-d958-4a7e-a340-bf9735e9edcb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:24.022541 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af849359-d958-4a7e-a340-bf9735e9edcb admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:24.041516 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1257/2484] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:47:24.051384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c63c56-dc98-4f09-a0d6-a6368a956a90 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:24.052810 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-14c63c56-dc98-4f09-a0d6-a6368a956a90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:24.107385 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1228/2485] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:47:23 2022] POST /identity/v3/projects => generated 367 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:47:24.111399 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af849359-d958-4a7e-a340-bf9735e9edcb admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:24.114986 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c63c56-dc98-4f09-a0d6-a6368a956a90 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:24.115473 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:24.147092 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:24.167449 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0aa549b-38e8-4454-8865-8e716085f54c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:24.168095 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0aa549b-38e8-4454-8865-8e716085f54c service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:24.168396 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0aa549b-38e8-4454-8865-8e716085f54c service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:24.168672 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0aa549b-38e8-4454-8865-8e716085f54c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:24.177178 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af849359-d958-4a7e-a340-bf9735e9edcb admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:24.177866 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0aa549b-38e8-4454-8865-8e716085f54c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:24.179393 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0aa549b-38e8-4454-8865-8e716085f54c service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:24.183119 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e26abb-8096-4052-8873-e36474e0bf24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:24.187958 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e26abb-8096-4052-8873-e36474e0bf24 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:24.187958 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e26abb-8096-4052-8873-e36474e0bf24 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:24.187958 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e26abb-8096-4052-8873-e36474e0bf24 admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:24.187958 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e26abb-8096-4052-8873-e36474e0bf24 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:24.187958 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e26abb-8096-4052-8873-e36474e0bf24 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:24.190692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1258/2486] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:47:24 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:47:24.274713 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1229/2487] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:47:24 2022] POST /identity/v3/users => generated 284 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:47:24.276249 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0aa549b-38e8-4454-8865-8e716085f54c service neutron] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:24.286458 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0aa549b-38e8-4454-8865-8e716085f54c service neutron] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:24.292150 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b75421-b34f-42c0-a94d-ed28978b4926 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:24.292819 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b75421-b34f-42c0-a94d-ed28978b4926 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:24.293112 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b75421-b34f-42c0-a94d-ed28978b4926 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:24.293396 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b75421-b34f-42c0-a94d-ed28978b4926 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:24.294131 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2b75421-b34f-42c0-a94d-ed28978b4926 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:24.295833 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2b75421-b34f-42c0-a94d-ed28978b4926 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:24.309958 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1259/2488] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:47:24 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:47:24.314211 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e26abb-8096-4052-8873-e36474e0bf24 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:24.325136 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e26abb-8096-4052-8873-e36474e0bf24 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:24.330227 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8aa22e0-2276-4e44-a22e-104a78078a51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:24.330982 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8aa22e0-2276-4e44-a22e-104a78078a51 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:24.331273 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8aa22e0-2276-4e44-a22e-104a78078a51 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:24.331583 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8aa22e0-2276-4e44-a22e-104a78078a51 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:24.332995 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8aa22e0-2276-4e44-a22e-104a78078a51 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:24.334063 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8aa22e0-2276-4e44-a22e-104a78078a51 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:24.345139 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1230/2489] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:47:24 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 12 17:47:24.349422 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b75421-b34f-42c0-a94d-ed28978b4926 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:24.360668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b75421-b34f-42c0-a94d-ed28978b4926 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:24.367811 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef115574-082f-492c-a27a-1a6b586c84e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:24.367811 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef115574-082f-492c-a27a-1a6b586c84e3 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:24.368157 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef115574-082f-492c-a27a-1a6b586c84e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:24.368157 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef115574-082f-492c-a27a-1a6b586c84e3 admin admin] PATH_INFO: `/v3/projects/8afbaf89b07146bd83330353bf6485cd/users/ab8119fc170e47eb81c4b0363d453edf/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:24.389132 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef115574-082f-492c-a27a-1a6b586c84e3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8afbaf89b07146bd83330353bf6485cd, user_id=ab8119fc170e47eb81c4b0363d453edf, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:24.394299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef115574-082f-492c-a27a-1a6b586c84e3 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:24.500226 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1260/2490] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:47:24 2022] PUT /identity/v3/projects/8afbaf89b07146bd83330353bf6485cd/users/ab8119fc170e47eb81c4b0363d453edf/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:24.510924 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e520bd-d577-45ed-946a-e0e5e676c449 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:24.511224 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e520bd-d577-45ed-946a-e0e5e676c449 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:24.511514 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e520bd-d577-45ed-946a-e0e5e676c449 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:24.540707 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-96e520bd-d577-45ed-946a-e0e5e676c449 None None] MFA Rules not processed for user `ab8119fc170e47eb81c4b0363d453edf`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:24.541934 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-96e520bd-d577-45ed-946a-e0e5e676c449 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:24.609561 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1231/2491] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:24 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:47:24.622014 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef115574-082f-492c-a27a-1a6b586c84e3 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:24.623830 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-ef115574-082f-492c-a27a-1a6b586c84e3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:24.625565 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e520bd-d577-45ed-946a-e0e5e676c449 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:24.627117 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-96e520bd-d577-45ed-946a-e0e5e676c449 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:24.699091 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e520bd-d577-45ed-946a-e0e5e676c449 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:24.703871 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297fa5c1-dfe8-49cd-90f7-9ef5b31e9bfe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:24.704744 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297fa5c1-dfe8-49cd-90f7-9ef5b31e9bfe admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:24.705036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297fa5c1-dfe8-49cd-90f7-9ef5b31e9bfe admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:24.705323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-297fa5c1-dfe8-49cd-90f7-9ef5b31e9bfe admin admin] PATH_INFO: `/v3/projects/a5f461a4f9ca421ba6e503aa160570f2` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:24.706562 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-297fa5c1-dfe8-49cd-90f7-9ef5b31e9bfe admin admin] RBAC: Authorizing `identity:delete_project(project_id=a5f461a4f9ca421ba6e503aa160570f2)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:24.708499 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-297fa5c1-dfe8-49cd-90f7-9ef5b31e9bfe admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:24.722257 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:24.735241 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef115574-082f-492c-a27a-1a6b586c84e3 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:24.739080 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a3b2b9-a965-4b8c-a7fb-67521830c812 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:24.739720 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a3b2b9-a965-4b8c-a7fb-67521830c812 service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:24.740012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a3b2b9-a965-4b8c-a7fb-67521830c812 service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:24.740297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a3b2b9-a965-4b8c-a7fb-67521830c812 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:24.749121 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a3b2b9-a965-4b8c-a7fb-67521830c812 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:24.750574 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a3b2b9-a965-4b8c-a7fb-67521830c812 service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:24.760670 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1261/2492] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:47:24 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:47:24.844260 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-297fa5c1-dfe8-49cd-90f7-9ef5b31e9bfe admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a5f461a4f9ca421ba6e503aa160570f2', 'request_id': 'req-297fa5c1-dfe8-49cd-90f7-9ef5b31e9bfe'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:24.850728 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-297fa5c1-dfe8-49cd-90f7-9ef5b31e9bfe admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a5f461a4f9ca421ba6e503aa160570f2', 'request_id': 'req-297fa5c1-dfe8-49cd-90f7-9ef5b31e9bfe'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:24.858301 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-297fa5c1-dfe8-49cd-90f7-9ef5b31e9bfe admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a5f461a4f9ca421ba6e503aa160570f2', 'request_id': 'req-297fa5c1-dfe8-49cd-90f7-9ef5b31e9bfe'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:24.932872 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-297fa5c1-dfe8-49cd-90f7-9ef5b31e9bfe admin admin] The token cache is being invalidate because project a5f461a4f9ca421ba6e503aa160570f2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:47:24.933171 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-297fa5c1-dfe8-49cd-90f7-9ef5b31e9bfe admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:24.933823 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-297fa5c1-dfe8-49cd-90f7-9ef5b31e9bfe admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:24.935425 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1232/2493] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:47:24 2022] DELETE /identity/v3/projects/a5f461a4f9ca421ba6e503aa160570f2 => generated 0 bytes in 310 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:24.939491 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a3b2b9-a965-4b8c-a7fb-67521830c812 service cinder] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:24.951346 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a3b2b9-a965-4b8c-a7fb-67521830c812 service cinder] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:25.010671 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73192937-5b69-409d-bc79-22c0b2360ebd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:25.011559 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73192937-5b69-409d-bc79-22c0b2360ebd admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:25.011842 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73192937-5b69-409d-bc79-22c0b2360ebd admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:25.012120 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73192937-5b69-409d-bc79-22c0b2360ebd admin admin] PATH_INFO: `/v3/users/3a569e35fbea448fa651a885a390b594` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:25.013423 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73192937-5b69-409d-bc79-22c0b2360ebd admin admin] RBAC: Authorizing `identity:delete_user(user_id=3a569e35fbea448fa651a885a390b594)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:25.015276 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73192937-5b69-409d-bc79-22c0b2360ebd admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:25.039337 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=95,keystone:INSERT=1,keystone:DELETE=4 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:25.199512 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-73192937-5b69-409d-bc79-22c0b2360ebd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3a569e35fbea448fa651a885a390b594'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:25.200177 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-73192937-5b69-409d-bc79-22c0b2360ebd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3a569e35fbea448fa651a885a390b594'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:25.200828 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-73192937-5b69-409d-bc79-22c0b2360ebd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3a569e35fbea448fa651a885a390b594'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:25.209174 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-73192937-5b69-409d-bc79-22c0b2360ebd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3a569e35fbea448fa651a885a390b594'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:25.223731 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-73192937-5b69-409d-bc79-22c0b2360ebd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3a569e35fbea448fa651a885a390b594'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:25.289054 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1262/2494] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:47:24 2022] DELETE /identity/v3/users/3a569e35fbea448fa651a885a390b594 => generated 0 bytes in 350 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:25.740881 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297fa5c1-dfe8-49cd-90f7-9ef5b31e9bfe admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:25.751810 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-297fa5c1-dfe8-49cd-90f7-9ef5b31e9bfe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:25.760486 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0f64f9-1e71-43c0-916e-2952157bdbcb None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:25.761632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0f64f9-1e71-43c0-916e-2952157bdbcb None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:25.761928 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0f64f9-1e71-43c0-916e-2952157bdbcb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:25.766374 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-297fa5c1-dfe8-49cd-90f7-9ef5b31e9bfe admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:25.781371 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-3f0f64f9-1e71-43c0-916e-2952157bdbcb None None] MFA Rules not processed for user `bde6ee9df10146d98991b0d4bb37fc48`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:25.782634 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-3f0f64f9-1e71-43c0-916e-2952157bdbcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:25.831828 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ba73df-28f5-4240-95c8-b62c5009b4da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:25.832713 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ba73df-28f5-4240-95c8-b62c5009b4da admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:25.833001 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ba73df-28f5-4240-95c8-b62c5009b4da admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:25.833289 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ba73df-28f5-4240-95c8-b62c5009b4da admin admin] PATH_INFO: `/v3/projects/86a5feac6dcf4533a9dbefa617eaef06` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:25.834500 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ba73df-28f5-4240-95c8-b62c5009b4da admin admin] RBAC: Authorizing `identity:delete_project(project_id=86a5feac6dcf4533a9dbefa617eaef06)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:25.836458 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ba73df-28f5-4240-95c8-b62c5009b4da admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:25.842988 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1263/2495] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:25 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:47:25.952283 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-41ba73df-28f5-4240-95c8-b62c5009b4da admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '86a5feac6dcf4533a9dbefa617eaef06', 'request_id': 'req-41ba73df-28f5-4240-95c8-b62c5009b4da'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:25.959125 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-41ba73df-28f5-4240-95c8-b62c5009b4da admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '86a5feac6dcf4533a9dbefa617eaef06', 'request_id': 'req-41ba73df-28f5-4240-95c8-b62c5009b4da'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:25.965704 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-41ba73df-28f5-4240-95c8-b62c5009b4da admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '86a5feac6dcf4533a9dbefa617eaef06', 'request_id': 'req-41ba73df-28f5-4240-95c8-b62c5009b4da'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:26.052792 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-41ba73df-28f5-4240-95c8-b62c5009b4da admin admin] The token cache is being invalidate because project 86a5feac6dcf4533a9dbefa617eaef06 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:47:26.053119 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-41ba73df-28f5-4240-95c8-b62c5009b4da admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:26.053785 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-41ba73df-28f5-4240-95c8-b62c5009b4da admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:26.055474 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1233/2496] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:47:25 2022] DELETE /identity/v3/projects/86a5feac6dcf4533a9dbefa617eaef06 => generated 0 bytes in 315 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:26.060522 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cae075d-8392-4b41-9c95-7409140bdd63 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:26.060822 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cae075d-8392-4b41-9c95-7409140bdd63 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:26.061134 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cae075d-8392-4b41-9c95-7409140bdd63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:26.081135 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-0cae075d-8392-4b41-9c95-7409140bdd63 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:26.082363 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-0cae075d-8392-4b41-9c95-7409140bdd63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:26.147192 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1264/2497] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:47:26.155714 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ca76c2-2bad-4d02-8b2e-d4f1d8455017 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:26.156160 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ca76c2-2bad-4d02-8b2e-d4f1d8455017 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:26.156520 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ca76c2-2bad-4d02-8b2e-d4f1d8455017 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:26.177003 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-42ca76c2-2bad-4d02-8b2e-d4f1d8455017 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:26.178273 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-42ca76c2-2bad-4d02-8b2e-d4f1d8455017 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:26.187094 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1234/2498] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:47:26.193229 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cae075d-8392-4b41-9c95-7409140bdd63 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:26.204083 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cae075d-8392-4b41-9c95-7409140bdd63 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:26.209188 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fd1729-3c68-41e4-be25-43b36426e896 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:26.209860 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fd1729-3c68-41e4-be25-43b36426e896 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:26.210150 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fd1729-3c68-41e4-be25-43b36426e896 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:26.210454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fd1729-3c68-41e4-be25-43b36426e896 admin admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:26.211207 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5fd1729-3c68-41e4-be25-43b36426e896 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:47:26.211521 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5fd1729-3c68-41e4-be25-43b36426e896 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:26.213996 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5fd1729-3c68-41e4-be25-43b36426e896 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:26.226519 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1265/2499] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:47:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:47:26.231712 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ca76c2-2bad-4d02-8b2e-d4f1d8455017 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:26.241227 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42ca76c2-2bad-4d02-8b2e-d4f1d8455017 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:26.246478 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff115ac8-166c-4c66-b92c-9a65d751e30d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:26.247148 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff115ac8-166c-4c66-b92c-9a65d751e30d admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:26.247432 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff115ac8-166c-4c66-b92c-9a65d751e30d admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:26.247847 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff115ac8-166c-4c66-b92c-9a65d751e30d admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:26.248456 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff115ac8-166c-4c66-b92c-9a65d751e30d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:26.250447 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff115ac8-166c-4c66-b92c-9a65d751e30d admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:26.352364 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1235/2500] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:47:26 2022] POST /identity/v3/projects => generated 387 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:47:26.356839 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fd1729-3c68-41e4-be25-43b36426e896 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:26.366960 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fd1729-3c68-41e4-be25-43b36426e896 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:26.405838 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:26.442977 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf778620-1a0a-4fdc-bd72-22fd58d8edae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:26.443919 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf778620-1a0a-4fdc-bd72-22fd58d8edae admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:26.444215 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf778620-1a0a-4fdc-bd72-22fd58d8edae admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:26.444500 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf778620-1a0a-4fdc-bd72-22fd58d8edae admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:26.445203 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf778620-1a0a-4fdc-bd72-22fd58d8edae admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:26.448601 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf778620-1a0a-4fdc-bd72-22fd58d8edae admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:26.556990 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1266/2501] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:47:26 2022] POST /identity/v3/users => generated 294 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:47:26.560971 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff115ac8-166c-4c66-b92c-9a65d751e30d admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:26.566863 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:26.573273 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff115ac8-166c-4c66-b92c-9a65d751e30d admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:26.579253 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1922a6fd-eb84-409f-b8a0-04a99c1fb3c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:26.579800 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1922a6fd-eb84-409f-b8a0-04a99c1fb3c2 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:26.580148 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1922a6fd-eb84-409f-b8a0-04a99c1fb3c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:26.580942 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1922a6fd-eb84-409f-b8a0-04a99c1fb3c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:26.582100 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1922a6fd-eb84-409f-b8a0-04a99c1fb3c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:26.583079 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1922a6fd-eb84-409f-b8a0-04a99c1fb3c2 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:26.594798 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1236/2502] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:47:26 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:47:26.598801 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf778620-1a0a-4fdc-bd72-22fd58d8edae admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:26.609032 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf778620-1a0a-4fdc-bd72-22fd58d8edae admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:26.613603 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d3b45e2-12a5-41fe-9dc4-8009753e0d90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:26.614255 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3b45e2-12a5-41fe-9dc4-8009753e0d90 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:26.614574 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3b45e2-12a5-41fe-9dc4-8009753e0d90 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:26.614856 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3b45e2-12a5-41fe-9dc4-8009753e0d90 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:26.615616 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d3b45e2-12a5-41fe-9dc4-8009753e0d90 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:26.617197 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d3b45e2-12a5-41fe-9dc4-8009753e0d90 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:26.628793 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1267/2503] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:47:26 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:47:26.633216 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1922a6fd-eb84-409f-b8a0-04a99c1fb3c2 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:26.644130 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1922a6fd-eb84-409f-b8a0-04a99c1fb3c2 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:26.648729 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ff4c52-bba5-4702-a0b7-8831b131391a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:26.649624 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ff4c52-bba5-4702-a0b7-8831b131391a admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:26.649909 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ff4c52-bba5-4702-a0b7-8831b131391a admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:26.650199 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ff4c52-bba5-4702-a0b7-8831b131391a admin admin] PATH_INFO: `/v3/projects/f363a639fe7046809ef891952cc0e94f/users/a49ddb827caf4a4085a67d3e4e1344e9/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:26.668475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ff4c52-bba5-4702-a0b7-8831b131391a admin admin] RBAC: Authorizing `identity:create_grant(project_id=f363a639fe7046809ef891952cc0e94f, user_id=a49ddb827caf4a4085a67d3e4e1344e9, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:26.671381 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ff4c52-bba5-4702-a0b7-8831b131391a admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:26.752637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1237/2504] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:47:26 2022] PUT /identity/v3/projects/f363a639fe7046809ef891952cc0e94f/users/a49ddb827caf4a4085a67d3e4e1344e9/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:26.764783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8595c5-9ec6-4d69-9d84-13ba07d66070 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:26.765251 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8595c5-9ec6-4d69-9d84-13ba07d66070 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:26.765338 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8595c5-9ec6-4d69-9d84-13ba07d66070 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:26.798824 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-3b8595c5-9ec6-4d69-9d84-13ba07d66070 None None] MFA Rules not processed for user `a49ddb827caf4a4085a67d3e4e1344e9`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:26.800109 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-3b8595c5-9ec6-4d69-9d84-13ba07d66070 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:26.881682 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1268/2505] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:26 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:47:26.887074 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ff4c52-bba5-4702-a0b7-8831b131391a admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:26.951539 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ff4c52-bba5-4702-a0b7-8831b131391a admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:26.956324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c464b22-32cc-4865-a614-13555334c6d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:26.956990 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c464b22-32cc-4865-a614-13555334c6d6 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:26.957281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c464b22-32cc-4865-a614-13555334c6d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:26.957564 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c464b22-32cc-4865-a614-13555334c6d6 admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:26.958260 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c464b22-32cc-4865-a614-13555334c6d6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:26.960219 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c464b22-32cc-4865-a614-13555334c6d6 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:27.083500 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1238/2506] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:47:26 2022] POST /identity/v3/projects => generated 387 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:47:27.087702 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8595c5-9ec6-4d69-9d84-13ba07d66070 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:27.098841 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b8595c5-9ec6-4d69-9d84-13ba07d66070 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:27.157369 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7bc4cff-9ab0-429f-b604-ce20fdfe1a42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:27.158115 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bc4cff-9ab0-429f-b604-ce20fdfe1a42 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:27.158815 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bc4cff-9ab0-429f-b604-ce20fdfe1a42 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:27.162243 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bc4cff-9ab0-429f-b604-ce20fdfe1a42 admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:27.162243 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7bc4cff-9ab0-429f-b604-ce20fdfe1a42 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:27.162243 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7bc4cff-9ab0-429f-b604-ce20fdfe1a42 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:27.232170 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1269/2507] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:47:27 2022] POST /identity/v3/users => generated 300 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:47:27.236354 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c464b22-32cc-4865-a614-13555334c6d6 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:27.247992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c464b22-32cc-4865-a614-13555334c6d6 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:27.252860 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9e0a15-329f-422c-8854-712fd86bd5b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:27.258518 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9e0a15-329f-422c-8854-712fd86bd5b5 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:27.258518 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9e0a15-329f-422c-8854-712fd86bd5b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:27.258518 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9e0a15-329f-422c-8854-712fd86bd5b5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:27.258518 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb9e0a15-329f-422c-8854-712fd86bd5b5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:27.258518 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb9e0a15-329f-422c-8854-712fd86bd5b5 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:27.268335 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1239/2508] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:47:27 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:47:27.272471 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7bc4cff-9ab0-429f-b604-ce20fdfe1a42 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:27.282772 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7bc4cff-9ab0-429f-b604-ce20fdfe1a42 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:27.287720 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a146f2b-6b58-4e21-849f-88ab37e7dbd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:27.288669 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a146f2b-6b58-4e21-849f-88ab37e7dbd0 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:27.288960 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a146f2b-6b58-4e21-849f-88ab37e7dbd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:27.289245 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a146f2b-6b58-4e21-849f-88ab37e7dbd0 admin admin] PATH_INFO: `/v3/projects/3fc64268c4dd4074a0c380a78b3dfea4/users/6503696c256041a291f3830f973ac2bf/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:27.305178 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a146f2b-6b58-4e21-849f-88ab37e7dbd0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3fc64268c4dd4074a0c380a78b3dfea4, user_id=6503696c256041a291f3830f973ac2bf, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:27.314574 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a146f2b-6b58-4e21-849f-88ab37e7dbd0 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:27.393894 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1270/2509] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:47:27 2022] PUT /identity/v3/projects/3fc64268c4dd4074a0c380a78b3dfea4/users/6503696c256041a291f3830f973ac2bf/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:27.398014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9e0a15-329f-422c-8854-712fd86bd5b5 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:27.409303 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9e0a15-329f-422c-8854-712fd86bd5b5 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:27.467485 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f2779a-9c5a-4a50-825c-2bf4fc6d8948 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:27.468170 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f2779a-9c5a-4a50-825c-2bf4fc6d8948 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:27.468460 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f2779a-9c5a-4a50-825c-2bf4fc6d8948 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:27.468740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f2779a-9c5a-4a50-825c-2bf4fc6d8948 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:27.469480 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f2779a-9c5a-4a50-825c-2bf4fc6d8948 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:27.471116 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f2779a-9c5a-4a50-825c-2bf4fc6d8948 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:27.483569 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1240/2510] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:47:27 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:47:27.487075 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a146f2b-6b58-4e21-849f-88ab37e7dbd0 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:27.497528 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a146f2b-6b58-4e21-849f-88ab37e7dbd0 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:27.502219 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9f6265-cb5a-43c1-93be-490f63aeb33d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:27.503143 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9f6265-cb5a-43c1-93be-490f63aeb33d admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:27.503431 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9f6265-cb5a-43c1-93be-490f63aeb33d admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:27.503711 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9f6265-cb5a-43c1-93be-490f63aeb33d admin admin] PATH_INFO: `/v3/projects/3fc64268c4dd4074a0c380a78b3dfea4/users/6503696c256041a291f3830f973ac2bf/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:27.505675 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee9f6265-cb5a-43c1-93be-490f63aeb33d admin admin] RBAC: Authorizing `identity:create_grant(project_id=3fc64268c4dd4074a0c380a78b3dfea4, user_id=6503696c256041a291f3830f973ac2bf, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:27.508380 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee9f6265-cb5a-43c1-93be-490f63aeb33d admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:27.510826 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:27.519499 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1271/2511] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:47:27 2022] PUT /identity/v3/projects/3fc64268c4dd4074a0c380a78b3dfea4/users/6503696c256041a291f3830f973ac2bf/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:27.530279 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97c4727-89e8-4d72-ba9f-f79096ca431b None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:27.530612 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97c4727-89e8-4d72-ba9f-f79096ca431b None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:27.530902 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97c4727-89e8-4d72-ba9f-f79096ca431b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:27.558753 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-c97c4727-89e8-4d72-ba9f-f79096ca431b None None] MFA Rules not processed for user `6503696c256041a291f3830f973ac2bf`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:27.560003 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-c97c4727-89e8-4d72-ba9f-f79096ca431b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:27.593123 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:27.637496 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1241/2512] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:27 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:47:27.650469 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9f6265-cb5a-43c1-93be-490f63aeb33d admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:27.652357 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-ee9f6265-cb5a-43c1-93be-490f63aeb33d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:27.710676 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9f6265-cb5a-43c1-93be-490f63aeb33d admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:27.714895 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c307c0-183a-4626-9126-c0a0d650a1d1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:27.715542 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c307c0-183a-4626-9126-c0a0d650a1d1 service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:27.715870 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c307c0-183a-4626-9126-c0a0d650a1d1 service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:27.716156 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c307c0-183a-4626-9126-c0a0d650a1d1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:27.725520 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c307c0-183a-4626-9126-c0a0d650a1d1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:27.727109 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c307c0-183a-4626-9126-c0a0d650a1d1 service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:27.738339 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1272/2513] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:47:27 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:47:27.803602 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97c4727-89e8-4d72-ba9f-f79096ca431b None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:27.867782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97c4727-89e8-4d72-ba9f-f79096ca431b None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:27.872639 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318a402c-a895-49e2-80a6-43b5132e2d30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:27.873516 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318a402c-a895-49e2-80a6-43b5132e2d30 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:27.873806 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318a402c-a895-49e2-80a6-43b5132e2d30 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:27.874093 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318a402c-a895-49e2-80a6-43b5132e2d30 admin admin] PATH_INFO: `/v3/users/a49ddb827caf4a4085a67d3e4e1344e9` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:27.875357 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-318a402c-a895-49e2-80a6-43b5132e2d30 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a49ddb827caf4a4085a67d3e4e1344e9)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:27.877288 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-318a402c-a895-49e2-80a6-43b5132e2d30 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:28.079710 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-318a402c-a895-49e2-80a6-43b5132e2d30 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a49ddb827caf4a4085a67d3e4e1344e9'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:28.195913 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-318a402c-a895-49e2-80a6-43b5132e2d30 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a49ddb827caf4a4085a67d3e4e1344e9'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:28.197349 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-318a402c-a895-49e2-80a6-43b5132e2d30 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a49ddb827caf4a4085a67d3e4e1344e9'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:28.205458 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-318a402c-a895-49e2-80a6-43b5132e2d30 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a49ddb827caf4a4085a67d3e4e1344e9'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:28.217522 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-318a402c-a895-49e2-80a6-43b5132e2d30 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a49ddb827caf4a4085a67d3e4e1344e9'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:28.223748 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1242/2514] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:47:27 2022] DELETE /identity/v3/users/a49ddb827caf4a4085a67d3e4e1344e9 => generated 0 bytes in 421 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:47:28.228686 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c21bb6-f74c-496f-aece-5c179ba926bd None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:28.229199 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c21bb6-f74c-496f-aece-5c179ba926bd None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:28.229326 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c21bb6-f74c-496f-aece-5c179ba926bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:28.248400 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-f5c21bb6-f74c-496f-aece-5c179ba926bd None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:28.249599 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-f5c21bb6-f74c-496f-aece-5c179ba926bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:28.311832 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1273/2515] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:47:28.322603 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318a402c-a895-49e2-80a6-43b5132e2d30 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:28.324217 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-318a402c-a895-49e2-80a6-43b5132e2d30 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:28.373925 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:28.386934 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318a402c-a895-49e2-80a6-43b5132e2d30 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:28.392397 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad60935-d39a-4afc-932f-d00dea4d5b32 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:28.393091 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad60935-d39a-4afc-932f-d00dea4d5b32 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:28.393385 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad60935-d39a-4afc-932f-d00dea4d5b32 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:28.393790 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad60935-d39a-4afc-932f-d00dea4d5b32 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:28.404102 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bad60935-d39a-4afc-932f-d00dea4d5b32 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:28.405524 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bad60935-d39a-4afc-932f-d00dea4d5b32 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:28.415547 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1243/2516] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:47:28 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:47:28.854684 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c21bb6-f74c-496f-aece-5c179ba926bd None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:28.856192 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-f5c21bb6-f74c-496f-aece-5c179ba926bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:28.868500 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c21bb6-f74c-496f-aece-5c179ba926bd None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:28.873262 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c2bee8-5376-483e-915e-5e4c633c73cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:28.874131 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c2bee8-5376-483e-915e-5e4c633c73cd admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:28.874415 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c2bee8-5376-483e-915e-5e4c633c73cd admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:28.878475 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c2bee8-5376-483e-915e-5e4c633c73cd admin admin] PATH_INFO: `/v3/projects/f363a639fe7046809ef891952cc0e94f` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:28.878475 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c2bee8-5376-483e-915e-5e4c633c73cd admin admin] RBAC: Authorizing `identity:delete_project(project_id=f363a639fe7046809ef891952cc0e94f)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:28.878475 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c2bee8-5376-483e-915e-5e4c633c73cd admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:29.021196 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-63c2bee8-5376-483e-915e-5e4c633c73cd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f363a639fe7046809ef891952cc0e94f', 'request_id': 'req-63c2bee8-5376-483e-915e-5e4c633c73cd'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:29.028228 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-63c2bee8-5376-483e-915e-5e4c633c73cd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f363a639fe7046809ef891952cc0e94f', 'request_id': 'req-63c2bee8-5376-483e-915e-5e4c633c73cd'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:29.104961 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-63c2bee8-5376-483e-915e-5e4c633c73cd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f363a639fe7046809ef891952cc0e94f', 'request_id': 'req-63c2bee8-5376-483e-915e-5e4c633c73cd'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:29.134060 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-63c2bee8-5376-483e-915e-5e4c633c73cd admin admin] The token cache is being invalidate because project f363a639fe7046809ef891952cc0e94f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:47:29.134364 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-63c2bee8-5376-483e-915e-5e4c633c73cd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:29.137401 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-63c2bee8-5376-483e-915e-5e4c633c73cd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:29.137401 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1274/2517] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:47:28 2022] DELETE /identity/v3/projects/f363a639fe7046809ef891952cc0e94f => generated 0 bytes in 283 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:29.141208 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad60935-d39a-4afc-932f-d00dea4d5b32 service neutron] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:29.151864 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad60935-d39a-4afc-932f-d00dea4d5b32 service neutron] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:29.213072 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-929583d1-98fe-4d5c-a8c5-41e95ab008ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:29.213969 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929583d1-98fe-4d5c-a8c5-41e95ab008ef admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:29.214260 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929583d1-98fe-4d5c-a8c5-41e95ab008ef admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:29.214577 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929583d1-98fe-4d5c-a8c5-41e95ab008ef admin admin] PATH_INFO: `/v3/users/6503696c256041a291f3830f973ac2bf` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:29.215826 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-929583d1-98fe-4d5c-a8c5-41e95ab008ef admin admin] RBAC: Authorizing `identity:delete_user(user_id=6503696c256041a291f3830f973ac2bf)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:29.217786 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-929583d1-98fe-4d5c-a8c5-41e95ab008ef admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:29.404736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-929583d1-98fe-4d5c-a8c5-41e95ab008ef admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6503696c256041a291f3830f973ac2bf'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:29.465245 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-929583d1-98fe-4d5c-a8c5-41e95ab008ef admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6503696c256041a291f3830f973ac2bf'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:29.465863 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-929583d1-98fe-4d5c-a8c5-41e95ab008ef admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6503696c256041a291f3830f973ac2bf'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:29.473217 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-929583d1-98fe-4d5c-a8c5-41e95ab008ef admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6503696c256041a291f3830f973ac2bf'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:29.485479 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-929583d1-98fe-4d5c-a8c5-41e95ab008ef admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6503696c256041a291f3830f973ac2bf'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:29.491534 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1244/2518] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:47:29 2022] DELETE /identity/v3/users/6503696c256041a291f3830f973ac2bf => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:29.861831 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c2bee8-5376-483e-915e-5e4c633c73cd admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:29.863681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-63c2bee8-5376-483e-915e-5e4c633c73cd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:29.885168 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c2bee8-5376-483e-915e-5e4c633c73cd admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:29.906540 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:INSERT=2,keystone:SELECT=91,keystone:DELETE=7 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:29.947011 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f0ad90-5d5a-446b-9502-e9b3115b4dc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:29.948406 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f0ad90-5d5a-446b-9502-e9b3115b4dc2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:29.948695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f0ad90-5d5a-446b-9502-e9b3115b4dc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:29.949076 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f0ad90-5d5a-446b-9502-e9b3115b4dc2 admin admin] PATH_INFO: `/v3/projects/3fc64268c4dd4074a0c380a78b3dfea4` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:29.950206 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f0ad90-5d5a-446b-9502-e9b3115b4dc2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3fc64268c4dd4074a0c380a78b3dfea4)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:29.952157 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f0ad90-5d5a-446b-9502-e9b3115b4dc2 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:30.075116 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-a5f0ad90-5d5a-446b-9502-e9b3115b4dc2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3fc64268c4dd4074a0c380a78b3dfea4', 'request_id': 'req-a5f0ad90-5d5a-446b-9502-e9b3115b4dc2'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:30.083121 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-a5f0ad90-5d5a-446b-9502-e9b3115b4dc2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3fc64268c4dd4074a0c380a78b3dfea4', 'request_id': 'req-a5f0ad90-5d5a-446b-9502-e9b3115b4dc2'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:30.172092 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-a5f0ad90-5d5a-446b-9502-e9b3115b4dc2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3fc64268c4dd4074a0c380a78b3dfea4', 'request_id': 'req-a5f0ad90-5d5a-446b-9502-e9b3115b4dc2'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:30.185868 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-a5f0ad90-5d5a-446b-9502-e9b3115b4dc2 admin admin] The token cache is being invalidate because project 3fc64268c4dd4074a0c380a78b3dfea4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:47:30.186168 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-a5f0ad90-5d5a-446b-9502-e9b3115b4dc2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:30.186900 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-a5f0ad90-5d5a-446b-9502-e9b3115b4dc2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:30.190643 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1275/2519] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:47:29 2022] DELETE /identity/v3/projects/3fc64268c4dd4074a0c380a78b3dfea4 => generated 0 bytes in 328 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:30.193791 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fdb77f-0df5-4762-ab6a-b707915d2744 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:30.194497 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fdb77f-0df5-4762-ab6a-b707915d2744 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:30.194497 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fdb77f-0df5-4762-ab6a-b707915d2744 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:30.219880 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-e4fdb77f-0df5-4762-ab6a-b707915d2744 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:30.221263 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-e4fdb77f-0df5-4762-ab6a-b707915d2744 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:30.291538 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1245/2520] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:47:30.299312 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbd55ac-f727-41e2-bba9-5413db031be4 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:30.299654 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbd55ac-f727-41e2-bba9-5413db031be4 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:30.299888 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbd55ac-f727-41e2-bba9-5413db031be4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:30.322211 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-6fbd55ac-f727-41e2-bba9-5413db031be4 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:30.323292 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-6fbd55ac-f727-41e2-bba9-5413db031be4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:30.332067 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1276/2521] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:47:30.336449 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4fdb77f-0df5-4762-ab6a-b707915d2744 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:30.344286 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:30.350166 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4fdb77f-0df5-4762-ab6a-b707915d2744 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:30.355572 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ea8711-cf93-4044-8a08-0886a1af19d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:30.356373 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ea8711-cf93-4044-8a08-0886a1af19d2 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:30.356733 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ea8711-cf93-4044-8a08-0886a1af19d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:30.357081 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ea8711-cf93-4044-8a08-0886a1af19d2 admin admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:30.357889 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ea8711-cf93-4044-8a08-0886a1af19d2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:47:30.359080 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ea8711-cf93-4044-8a08-0886a1af19d2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:30.360324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ea8711-cf93-4044-8a08-0886a1af19d2 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:30.377385 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1246/2522] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:47:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:47:30.378630 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbd55ac-f727-41e2-bba9-5413db031be4 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:30.389459 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbd55ac-f727-41e2-bba9-5413db031be4 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:30.397272 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1695e749-3067-4c5e-bc90-b90ef8e3158d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:30.400982 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1695e749-3067-4c5e-bc90-b90ef8e3158d admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:30.400982 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1695e749-3067-4c5e-bc90-b90ef8e3158d admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:30.400982 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1695e749-3067-4c5e-bc90-b90ef8e3158d admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:30.400982 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1695e749-3067-4c5e-bc90-b90ef8e3158d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:30.401775 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1695e749-3067-4c5e-bc90-b90ef8e3158d admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:30.468563 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1277/2523] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:47:30 2022] POST /identity/v3/projects => generated 385 bytes in 89 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:47:30.474889 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ea8711-cf93-4044-8a08-0886a1af19d2 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:30.486042 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ea8711-cf93-4044-8a08-0886a1af19d2 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:30.549013 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27878504-8eb0-4985-9504-73feb3af96b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:30.549784 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27878504-8eb0-4985-9504-73feb3af96b5 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:30.550145 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27878504-8eb0-4985-9504-73feb3af96b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:30.550528 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27878504-8eb0-4985-9504-73feb3af96b5 admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:30.551367 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27878504-8eb0-4985-9504-73feb3af96b5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:30.553508 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27878504-8eb0-4985-9504-73feb3af96b5 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:30.670692 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1247/2524] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:47:30 2022] POST /identity/v3/users => generated 293 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:47:30.674931 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1695e749-3067-4c5e-bc90-b90ef8e3158d admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:30.685525 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1695e749-3067-4c5e-bc90-b90ef8e3158d admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:30.690724 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c791ff4-1d66-485b-86f5-5013ffb66abc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:30.691389 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c791ff4-1d66-485b-86f5-5013ffb66abc admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:30.691680 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c791ff4-1d66-485b-86f5-5013ffb66abc admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:30.691964 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c791ff4-1d66-485b-86f5-5013ffb66abc admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:30.692706 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c791ff4-1d66-485b-86f5-5013ffb66abc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:30.694646 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c791ff4-1d66-485b-86f5-5013ffb66abc admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:30.706926 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1278/2525] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:47:30 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:47:30.711122 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27878504-8eb0-4985-9504-73feb3af96b5 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:30.723150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27878504-8eb0-4985-9504-73feb3af96b5 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:30.733395 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71da9950-1eb1-444d-9387-4c0636843c66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:30.734183 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71da9950-1eb1-444d-9387-4c0636843c66 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:30.734504 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71da9950-1eb1-444d-9387-4c0636843c66 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:30.734812 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71da9950-1eb1-444d-9387-4c0636843c66 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:30.735617 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71da9950-1eb1-444d-9387-4c0636843c66 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:30.737482 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71da9950-1eb1-444d-9387-4c0636843c66 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:30.751875 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1248/2526] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:47:30 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 12 17:47:30.753129 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c791ff4-1d66-485b-86f5-5013ffb66abc admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:30.764268 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c791ff4-1d66-485b-86f5-5013ffb66abc admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:30.768880 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ed0b81-d1e9-4fdb-9c74-ea96da903e38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:30.769760 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ed0b81-d1e9-4fdb-9c74-ea96da903e38 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:30.770046 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ed0b81-d1e9-4fdb-9c74-ea96da903e38 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:30.770325 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ed0b81-d1e9-4fdb-9c74-ea96da903e38 admin admin] PATH_INFO: `/v3/projects/dfa2e8f4698b4e1bb9beac6c8ba85083/users/43f0d4a573ba4fd59666d1b131d62779/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:30.786783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ed0b81-d1e9-4fdb-9c74-ea96da903e38 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dfa2e8f4698b4e1bb9beac6c8ba85083, user_id=43f0d4a573ba4fd59666d1b131d62779, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:30.789584 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ed0b81-d1e9-4fdb-9c74-ea96da903e38 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:30.859849 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1279/2527] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:47:30 2022] PUT /identity/v3/projects/dfa2e8f4698b4e1bb9beac6c8ba85083/users/43f0d4a573ba4fd59666d1b131d62779/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:30.931391 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be0beff-f67f-4be8-b441-26f039dd9419 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:30.931737 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be0beff-f67f-4be8-b441-26f039dd9419 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:30.932269 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be0beff-f67f-4be8-b441-26f039dd9419 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:30.960438 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-1be0beff-f67f-4be8-b441-26f039dd9419 None None] MFA Rules not processed for user `43f0d4a573ba4fd59666d1b131d62779`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:30.960944 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-1be0beff-f67f-4be8-b441-26f039dd9419 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:31.026701 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1249/2528] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:30 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:47:31.031225 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ed0b81-d1e9-4fdb-9c74-ea96da903e38 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:31.080460 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:31.094480 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ed0b81-d1e9-4fdb-9c74-ea96da903e38 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:31.099509 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ac4821-05aa-49fe-a657-8bab9c705959 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:31.100188 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ac4821-05aa-49fe-a657-8bab9c705959 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:31.100467 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ac4821-05aa-49fe-a657-8bab9c705959 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:31.100747 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ac4821-05aa-49fe-a657-8bab9c705959 admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:31.101430 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ac4821-05aa-49fe-a657-8bab9c705959 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:31.103415 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ac4821-05aa-49fe-a657-8bab9c705959 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:31.195605 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1280/2529] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:47:31 2022] POST /identity/v3/projects => generated 387 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:47:31.196760 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be0beff-f67f-4be8-b441-26f039dd9419 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:31.208103 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be0beff-f67f-4be8-b441-26f039dd9419 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:31.267315 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d118d645-b8ca-4929-afa3-a36583472f74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:31.268282 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d118d645-b8ca-4929-afa3-a36583472f74 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:31.268573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d118d645-b8ca-4929-afa3-a36583472f74 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:31.268856 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d118d645-b8ca-4929-afa3-a36583472f74 admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:31.269558 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d118d645-b8ca-4929-afa3-a36583472f74 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:31.271538 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d118d645-b8ca-4929-afa3-a36583472f74 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:31.278323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:31.345599 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1250/2530] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:47:31 2022] POST /identity/v3/users => generated 294 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:47:31.349743 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ac4821-05aa-49fe-a657-8bab9c705959 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:31.360935 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ac4821-05aa-49fe-a657-8bab9c705959 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:31.365881 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5f58f3-b175-440e-863d-d8f48408c27d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:31.366599 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5f58f3-b175-440e-863d-d8f48408c27d admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:31.366894 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5f58f3-b175-440e-863d-d8f48408c27d admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:31.367180 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5f58f3-b175-440e-863d-d8f48408c27d admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:31.367924 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5f58f3-b175-440e-863d-d8f48408c27d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:31.370184 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e5f58f3-b175-440e-863d-d8f48408c27d admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:31.382541 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1281/2531] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:47:31 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:47:31.387278 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d118d645-b8ca-4929-afa3-a36583472f74 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:31.398558 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d118d645-b8ca-4929-afa3-a36583472f74 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:31.403938 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021c9ac0-99f9-4b2c-bdc2-d48d9a7495ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:31.406321 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021c9ac0-99f9-4b2c-bdc2-d48d9a7495ae admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:31.406321 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021c9ac0-99f9-4b2c-bdc2-d48d9a7495ae admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:31.406321 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021c9ac0-99f9-4b2c-bdc2-d48d9a7495ae admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:31.406321 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021c9ac0-99f9-4b2c-bdc2-d48d9a7495ae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:31.407540 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021c9ac0-99f9-4b2c-bdc2-d48d9a7495ae admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:31.418313 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1251/2532] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:47:31 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:47:31.422714 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5f58f3-b175-440e-863d-d8f48408c27d admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:31.433492 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5f58f3-b175-440e-863d-d8f48408c27d admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:31.442669 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cda4c28-03d4-461f-871d-c46107025601 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:31.443848 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cda4c28-03d4-461f-871d-c46107025601 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:31.444134 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cda4c28-03d4-461f-871d-c46107025601 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:31.444417 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cda4c28-03d4-461f-871d-c46107025601 admin admin] PATH_INFO: `/v3/projects/2ccc5363c96e4eaf96ff870dea7bd647/users/59a43d118c004751989f8eec8d38728c/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:31.461589 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cda4c28-03d4-461f-871d-c46107025601 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2ccc5363c96e4eaf96ff870dea7bd647, user_id=59a43d118c004751989f8eec8d38728c, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:31.464420 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cda4c28-03d4-461f-871d-c46107025601 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:31.569534 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1282/2533] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:47:31 2022] PUT /identity/v3/projects/2ccc5363c96e4eaf96ff870dea7bd647/users/59a43d118c004751989f8eec8d38728c/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:31.580935 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8338387-9e2e-425e-a988-63acc123bd7d None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:31.581624 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8338387-9e2e-425e-a988-63acc123bd7d None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:31.582079 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8338387-9e2e-425e-a988-63acc123bd7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:31.614570 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-f8338387-9e2e-425e-a988-63acc123bd7d None None] MFA Rules not processed for user `59a43d118c004751989f8eec8d38728c`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:31.615820 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-f8338387-9e2e-425e-a988-63acc123bd7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:31.700239 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1252/2534] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:31 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:47:31.704718 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cda4c28-03d4-461f-871d-c46107025601 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:31.768821 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cda4c28-03d4-461f-871d-c46107025601 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:31.773601 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31aade30-a50b-401d-8b73-8241128e2ce8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:31.774273 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31aade30-a50b-401d-8b73-8241128e2ce8 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:31.774599 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31aade30-a50b-401d-8b73-8241128e2ce8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:31.774882 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31aade30-a50b-401d-8b73-8241128e2ce8 admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:31.776197 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31aade30-a50b-401d-8b73-8241128e2ce8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:31.778143 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31aade30-a50b-401d-8b73-8241128e2ce8 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:31.861349 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1283/2535] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:47:31 2022] POST /identity/v3/projects => generated 387 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:47:31.865299 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8338387-9e2e-425e-a988-63acc123bd7d None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:31.876718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8338387-9e2e-425e-a988-63acc123bd7d None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:31.937482 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f019dc3-b4aa-463a-8e4d-1476b5b420dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:31.938210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f019dc3-b4aa-463a-8e4d-1476b5b420dd admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:31.938530 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f019dc3-b4aa-463a-8e4d-1476b5b420dd admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:31.938813 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f019dc3-b4aa-463a-8e4d-1476b5b420dd admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:31.939527 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f019dc3-b4aa-463a-8e4d-1476b5b420dd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:31.941464 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f019dc3-b4aa-463a-8e4d-1476b5b420dd admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:32.002958 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1253/2536] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:47:31 2022] POST /identity/v3/users => generated 300 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:47:32.006599 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31aade30-a50b-401d-8b73-8241128e2ce8 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:32.017840 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31aade30-a50b-401d-8b73-8241128e2ce8 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:32.022526 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0fa513-8a08-446c-8131-ddc4de0de2d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:32.023189 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0fa513-8a08-446c-8131-ddc4de0de2d1 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:32.023549 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0fa513-8a08-446c-8131-ddc4de0de2d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:32.023766 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0fa513-8a08-446c-8131-ddc4de0de2d1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:32.024500 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0fa513-8a08-446c-8131-ddc4de0de2d1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:32.026202 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0fa513-8a08-446c-8131-ddc4de0de2d1 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:32.036842 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1284/2537] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:47:32 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:47:32.041051 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f019dc3-b4aa-463a-8e4d-1476b5b420dd admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:32.051071 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f019dc3-b4aa-463a-8e4d-1476b5b420dd admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:32.055641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-686bc6cc-c16d-4513-b0f1-a6c339c6bf6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:32.056515 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686bc6cc-c16d-4513-b0f1-a6c339c6bf6e admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:32.057298 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686bc6cc-c16d-4513-b0f1-a6c339c6bf6e admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:32.057578 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686bc6cc-c16d-4513-b0f1-a6c339c6bf6e admin admin] PATH_INFO: `/v3/projects/67749de63a36467f86a5c66292912a32/users/e66371b16f464f6faaa0a16157cfd6ee/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:32.072315 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-686bc6cc-c16d-4513-b0f1-a6c339c6bf6e admin admin] RBAC: Authorizing `identity:create_grant(project_id=67749de63a36467f86a5c66292912a32, user_id=e66371b16f464f6faaa0a16157cfd6ee, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:32.075715 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-686bc6cc-c16d-4513-b0f1-a6c339c6bf6e admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:32.140419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1254/2538] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:47:32 2022] PUT /identity/v3/projects/67749de63a36467f86a5c66292912a32/users/e66371b16f464f6faaa0a16157cfd6ee/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:32.144599 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0fa513-8a08-446c-8131-ddc4de0de2d1 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:32.155936 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0fa513-8a08-446c-8131-ddc4de0de2d1 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:32.215288 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b59a8786-ecd4-4f5b-a426-7433a01b2762 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:32.215961 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59a8786-ecd4-4f5b-a426-7433a01b2762 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:32.216253 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59a8786-ecd4-4f5b-a426-7433a01b2762 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:32.216537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59a8786-ecd4-4f5b-a426-7433a01b2762 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:32.217272 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b59a8786-ecd4-4f5b-a426-7433a01b2762 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:32.219003 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b59a8786-ecd4-4f5b-a426-7433a01b2762 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:32.229897 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1285/2539] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:47:32 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:47:32.234311 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-686bc6cc-c16d-4513-b0f1-a6c339c6bf6e admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:32.245739 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-686bc6cc-c16d-4513-b0f1-a6c339c6bf6e admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:32.250419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eca2981-fa09-4bfe-aeaa-5c03cfc36491 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:32.251363 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eca2981-fa09-4bfe-aeaa-5c03cfc36491 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:32.251650 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eca2981-fa09-4bfe-aeaa-5c03cfc36491 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:32.251969 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eca2981-fa09-4bfe-aeaa-5c03cfc36491 admin admin] PATH_INFO: `/v3/projects/67749de63a36467f86a5c66292912a32/users/e66371b16f464f6faaa0a16157cfd6ee/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:32.253918 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eca2981-fa09-4bfe-aeaa-5c03cfc36491 admin admin] RBAC: Authorizing `identity:create_grant(project_id=67749de63a36467f86a5c66292912a32, user_id=e66371b16f464f6faaa0a16157cfd6ee, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:32.256684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eca2981-fa09-4bfe-aeaa-5c03cfc36491 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:32.267189 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1255/2540] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:47:32 2022] PUT /identity/v3/projects/67749de63a36467f86a5c66292912a32/users/e66371b16f464f6faaa0a16157cfd6ee/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:32.277586 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c375b6c-3e4c-4b0a-8da2-8901c2a3488f None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:32.277883 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c375b6c-3e4c-4b0a-8da2-8901c2a3488f None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:32.278199 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c375b6c-3e4c-4b0a-8da2-8901c2a3488f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:32.290833 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:32.307299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-6c375b6c-3e4c-4b0a-8da2-8901c2a3488f None None] MFA Rules not processed for user `e66371b16f464f6faaa0a16157cfd6ee`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:32.308629 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-6c375b6c-3e4c-4b0a-8da2-8901c2a3488f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:32.381893 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1286/2541] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:32 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:47:32.400943 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eca2981-fa09-4bfe-aeaa-5c03cfc36491 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:32.400943 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-4eca2981-fa09-4bfe-aeaa-5c03cfc36491 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:32.425294 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:INSERT=8,keystone:SELECT=92 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:32.460829 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eca2981-fa09-4bfe-aeaa-5c03cfc36491 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:32.465160 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f223937d-1bed-463d-945f-f48bf9345caa service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:32.465815 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f223937d-1bed-463d-945f-f48bf9345caa service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:32.466106 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f223937d-1bed-463d-945f-f48bf9345caa service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:32.466392 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f223937d-1bed-463d-945f-f48bf9345caa service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:32.475767 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f223937d-1bed-463d-945f-f48bf9345caa service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:32.477189 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f223937d-1bed-463d-945f-f48bf9345caa service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:32.487828 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1256/2542] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:47:32 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:47:33.634524 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c375b6c-3e4c-4b0a-8da2-8901c2a3488f None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:33.636247 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-6c375b6c-3e4c-4b0a-8da2-8901c2a3488f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:33.649281 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c375b6c-3e4c-4b0a-8da2-8901c2a3488f None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:33.653299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37ec752-307f-4ac6-a70b-4bbaca731034 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:33.653952 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37ec752-307f-4ac6-a70b-4bbaca731034 service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:33.654243 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37ec752-307f-4ac6-a70b-4bbaca731034 service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:33.654563 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37ec752-307f-4ac6-a70b-4bbaca731034 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:33.715728 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f37ec752-307f-4ac6-a70b-4bbaca731034 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:33.717184 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f37ec752-307f-4ac6-a70b-4bbaca731034 service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:33.726693 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1287/2543] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:47:33 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:47:34.752374 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f223937d-1bed-463d-945f-f48bf9345caa service cinder] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:34.754203 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-f223937d-1bed-463d-945f-f48bf9345caa service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:34.830652 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f223937d-1bed-463d-945f-f48bf9345caa service cinder] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:34.835209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e250024-1d6f-42ba-a087-511b891fe39f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:34.836095 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e250024-1d6f-42ba-a087-511b891fe39f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:34.836384 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e250024-1d6f-42ba-a087-511b891fe39f admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:34.836663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e250024-1d6f-42ba-a087-511b891fe39f admin admin] PATH_INFO: `/v3/users/ab8119fc170e47eb81c4b0363d453edf` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:34.837866 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e250024-1d6f-42ba-a087-511b891fe39f admin admin] RBAC: Authorizing `identity:delete_user(user_id=ab8119fc170e47eb81c4b0363d453edf)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:34.839912 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e250024-1d6f-42ba-a087-511b891fe39f admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:35.078781 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-5e250024-1d6f-42ba-a087-511b891fe39f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ab8119fc170e47eb81c4b0363d453edf'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:35.196321 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-5e250024-1d6f-42ba-a087-511b891fe39f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ab8119fc170e47eb81c4b0363d453edf'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:35.197020 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-5e250024-1d6f-42ba-a087-511b891fe39f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ab8119fc170e47eb81c4b0363d453edf'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:35.206180 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-5e250024-1d6f-42ba-a087-511b891fe39f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ab8119fc170e47eb81c4b0363d453edf'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:35.222161 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-5e250024-1d6f-42ba-a087-511b891fe39f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ab8119fc170e47eb81c4b0363d453edf'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:35.229327 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1257/2544] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:47:34 2022] DELETE /identity/v3/users/ab8119fc170e47eb81c4b0363d453edf => generated 0 bytes in 478 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:35.240737 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6afa190-4859-4ec4-a5b0-6c8a0ba07a88 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:35.241226 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6afa190-4859-4ec4-a5b0-6c8a0ba07a88 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:35.241546 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6afa190-4859-4ec4-a5b0-6c8a0ba07a88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:35.263200 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-e6afa190-4859-4ec4-a5b0-6c8a0ba07a88 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:35.264467 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-e6afa190-4859-4ec4-a5b0-6c8a0ba07a88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:35.333822 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1288/2545] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:47:35.344016 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e250024-1d6f-42ba-a087-511b891fe39f admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:35.345654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-5e250024-1d6f-42ba-a087-511b891fe39f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:35.395681 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:35.408170 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e250024-1d6f-42ba-a087-511b891fe39f admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:35.411980 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-765b420b-4f77-455f-9884-ca37c6ba8ecb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:35.412647 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765b420b-4f77-455f-9884-ca37c6ba8ecb service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:35.412952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765b420b-4f77-455f-9884-ca37c6ba8ecb service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:35.413228 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765b420b-4f77-455f-9884-ca37c6ba8ecb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:35.426066 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-765b420b-4f77-455f-9884-ca37c6ba8ecb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:35.426066 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-765b420b-4f77-455f-9884-ca37c6ba8ecb service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:35.436514 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1258/2546] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:47:35 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:47:35.981094 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6afa190-4859-4ec4-a5b0-6c8a0ba07a88 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:35.984059 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-e6afa190-4859-4ec4-a5b0-6c8a0ba07a88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:36.001984 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6afa190-4859-4ec4-a5b0-6c8a0ba07a88 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:36.013177 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d93fd0-0718-4c6b-bb54-a187acb2bee6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:36.014174 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d93fd0-0718-4c6b-bb54-a187acb2bee6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:36.014679 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d93fd0-0718-4c6b-bb54-a187acb2bee6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:36.015135 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d93fd0-0718-4c6b-bb54-a187acb2bee6 admin admin] PATH_INFO: `/v3/projects/8afbaf89b07146bd83330353bf6485cd` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:36.017816 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d93fd0-0718-4c6b-bb54-a187acb2bee6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8afbaf89b07146bd83330353bf6485cd)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:36.025241 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68d93fd0-0718-4c6b-bb54-a187acb2bee6 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:36.146993 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:36.166110 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-68d93fd0-0718-4c6b-bb54-a187acb2bee6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8afbaf89b07146bd83330353bf6485cd', 'request_id': 'req-68d93fd0-0718-4c6b-bb54-a187acb2bee6'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:36.173227 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-68d93fd0-0718-4c6b-bb54-a187acb2bee6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8afbaf89b07146bd83330353bf6485cd', 'request_id': 'req-68d93fd0-0718-4c6b-bb54-a187acb2bee6'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:36.257624 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-68d93fd0-0718-4c6b-bb54-a187acb2bee6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8afbaf89b07146bd83330353bf6485cd', 'request_id': 'req-68d93fd0-0718-4c6b-bb54-a187acb2bee6'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:36.272436 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-68d93fd0-0718-4c6b-bb54-a187acb2bee6 admin admin] The token cache is being invalidate because project 8afbaf89b07146bd83330353bf6485cd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:47:36.272736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-68d93fd0-0718-4c6b-bb54-a187acb2bee6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:36.273393 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-68d93fd0-0718-4c6b-bb54-a187acb2bee6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:36.275010 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1289/2547] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:47:35 2022] DELETE /identity/v3/projects/8afbaf89b07146bd83330353bf6485cd => generated 0 bytes in 295 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:36.281515 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2743d3b4-bb6c-44c6-9f19-53b92c3e2199 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:36.282025 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2743d3b4-bb6c-44c6-9f19-53b92c3e2199 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:36.282357 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2743d3b4-bb6c-44c6-9f19-53b92c3e2199 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:36.306134 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-2743d3b4-bb6c-44c6-9f19-53b92c3e2199 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:36.307493 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-2743d3b4-bb6c-44c6-9f19-53b92c3e2199 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:36.384448 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1259/2548] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:47:36.395307 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985e62e6-98a0-4838-b015-b198211d81b5 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:36.395603 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985e62e6-98a0-4838-b015-b198211d81b5 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:36.395942 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985e62e6-98a0-4838-b015-b198211d81b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:36.423885 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-985e62e6-98a0-4838-b015-b198211d81b5 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:36.425134 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-985e62e6-98a0-4838-b015-b198211d81b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:36.433579 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1290/2549] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:47:36.437952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2743d3b4-bb6c-44c6-9f19-53b92c3e2199 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:36.448512 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2743d3b4-bb6c-44c6-9f19-53b92c3e2199 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:36.453431 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe15d79a-013f-47d1-9c7b-d295c2fddc83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:36.454095 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe15d79a-013f-47d1-9c7b-d295c2fddc83 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:36.454381 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe15d79a-013f-47d1-9c7b-d295c2fddc83 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:36.454700 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe15d79a-013f-47d1-9c7b-d295c2fddc83 admin admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:36.455430 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe15d79a-013f-47d1-9c7b-d295c2fddc83 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:47:36.455741 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe15d79a-013f-47d1-9c7b-d295c2fddc83 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:36.457332 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe15d79a-013f-47d1-9c7b-d295c2fddc83 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:36.473505 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1260/2550] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:47:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:47:36.474714 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985e62e6-98a0-4838-b015-b198211d81b5 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:36.485140 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985e62e6-98a0-4838-b015-b198211d81b5 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:36.490043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b08de0b-3bc6-4f78-9df7-d652d7f6eab8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:36.490871 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b08de0b-3bc6-4f78-9df7-d652d7f6eab8 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:36.491224 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b08de0b-3bc6-4f78-9df7-d652d7f6eab8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:36.491506 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b08de0b-3bc6-4f78-9df7-d652d7f6eab8 admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:36.492201 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b08de0b-3bc6-4f78-9df7-d652d7f6eab8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:36.494134 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b08de0b-3bc6-4f78-9df7-d652d7f6eab8 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:36.637230 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1291/2551] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:47:36 2022] POST /identity/v3/projects => generated 373 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:47:36.642035 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe15d79a-013f-47d1-9c7b-d295c2fddc83 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:36.653568 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe15d79a-013f-47d1-9c7b-d295c2fddc83 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:36.712792 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9d5c7f-fe46-4a12-8108-4433fd1e87a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:36.713524 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9d5c7f-fe46-4a12-8108-4433fd1e87a8 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:36.713809 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9d5c7f-fe46-4a12-8108-4433fd1e87a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:36.714090 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9d5c7f-fe46-4a12-8108-4433fd1e87a8 admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:36.715728 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9d5c7f-fe46-4a12-8108-4433fd1e87a8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:36.717761 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9d5c7f-fe46-4a12-8108-4433fd1e87a8 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:36.819243 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1261/2552] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:47:36 2022] POST /identity/v3/users => generated 287 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:47:36.823193 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b08de0b-3bc6-4f78-9df7-d652d7f6eab8 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:36.834815 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b08de0b-3bc6-4f78-9df7-d652d7f6eab8 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:36.840038 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df8d17db-50bc-4df2-9151-c7af92c1d8bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:36.840867 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8d17db-50bc-4df2-9151-c7af92c1d8bc admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:36.841197 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8d17db-50bc-4df2-9151-c7af92c1d8bc admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:36.841486 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8d17db-50bc-4df2-9151-c7af92c1d8bc admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:36.844958 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df8d17db-50bc-4df2-9151-c7af92c1d8bc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:36.844958 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df8d17db-50bc-4df2-9151-c7af92c1d8bc admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:36.855042 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1292/2553] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:47:36 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:47:36.859204 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9d5c7f-fe46-4a12-8108-4433fd1e87a8 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:36.869774 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9d5c7f-fe46-4a12-8108-4433fd1e87a8 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:36.876226 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f1064c-9561-4b62-8078-f893ecbf513b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:36.876918 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f1064c-9561-4b62-8078-f893ecbf513b admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:36.877209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f1064c-9561-4b62-8078-f893ecbf513b admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:36.877489 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f1064c-9561-4b62-8078-f893ecbf513b admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:36.878251 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4f1064c-9561-4b62-8078-f893ecbf513b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:36.880739 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4f1064c-9561-4b62-8078-f893ecbf513b admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:36.894517 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1262/2554] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:47:36 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 12 17:47:36.898856 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df8d17db-50bc-4df2-9151-c7af92c1d8bc admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:36.910281 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df8d17db-50bc-4df2-9151-c7af92c1d8bc admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:36.915509 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc157eca-316d-440a-8851-1f71b86762b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:36.916550 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc157eca-316d-440a-8851-1f71b86762b8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:36.916906 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc157eca-316d-440a-8851-1f71b86762b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:36.917626 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc157eca-316d-440a-8851-1f71b86762b8 admin admin] PATH_INFO: `/v3/projects/06574e6e01a24509affb68857311b6f1/users/a150dc5df85048a28450e7f7311fba0d/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:36.937041 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc157eca-316d-440a-8851-1f71b86762b8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=06574e6e01a24509affb68857311b6f1, user_id=a150dc5df85048a28450e7f7311fba0d, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:36.939808 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc157eca-316d-440a-8851-1f71b86762b8 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:37.015366 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1293/2555] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:47:36 2022] PUT /identity/v3/projects/06574e6e01a24509affb68857311b6f1/users/a150dc5df85048a28450e7f7311fba0d/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:37.026271 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30f456d-9e97-41e3-925b-015eaa08fa06 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:37.026816 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30f456d-9e97-41e3-925b-015eaa08fa06 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:37.027713 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e30f456d-9e97-41e3-925b-015eaa08fa06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:37.057516 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-e30f456d-9e97-41e3-925b-015eaa08fa06 None None] MFA Rules not processed for user `a150dc5df85048a28450e7f7311fba0d`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:37.059310 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-e30f456d-9e97-41e3-925b-015eaa08fa06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:37.106786 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:37.133621 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1263/2556] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:37 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:47:37.146424 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc157eca-316d-440a-8851-1f71b86762b8 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:37.148455 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-cc157eca-316d-440a-8851-1f71b86762b8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:37.238852 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc157eca-316d-440a-8851-1f71b86762b8 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:37.247288 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0275941-1177-401d-b44f-c7178ea973e7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:37.248089 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0275941-1177-401d-b44f-c7178ea973e7 service nova] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:37.248839 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0275941-1177-401d-b44f-c7178ea973e7 service nova] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:37.248839 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0275941-1177-401d-b44f-c7178ea973e7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:37.258357 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0275941-1177-401d-b44f-c7178ea973e7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:37.260424 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0275941-1177-401d-b44f-c7178ea973e7 service nova] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:37.272807 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1294/2557] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:47:37 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:47:37.523582 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30f456d-9e97-41e3-925b-015eaa08fa06 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:37.594115 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e30f456d-9e97-41e3-925b-015eaa08fa06 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:37.599022 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a353576-f5bb-4988-a357-653b81797667 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:37.600326 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a353576-f5bb-4988-a357-653b81797667 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:37.600628 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a353576-f5bb-4988-a357-653b81797667 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:37.600916 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a353576-f5bb-4988-a357-653b81797667 admin admin] PATH_INFO: `/v3/users/a150dc5df85048a28450e7f7311fba0d` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:37.602178 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a353576-f5bb-4988-a357-653b81797667 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a150dc5df85048a28450e7f7311fba0d)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:37.605815 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a353576-f5bb-4988-a357-653b81797667 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:37.830115 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-3a353576-f5bb-4988-a357-653b81797667 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a150dc5df85048a28450e7f7311fba0d'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:38.017829 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-3a353576-f5bb-4988-a357-653b81797667 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a150dc5df85048a28450e7f7311fba0d'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:38.021970 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-3a353576-f5bb-4988-a357-653b81797667 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a150dc5df85048a28450e7f7311fba0d'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:38.028354 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-3a353576-f5bb-4988-a357-653b81797667 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a150dc5df85048a28450e7f7311fba0d'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:38.041240 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-3a353576-f5bb-4988-a357-653b81797667 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a150dc5df85048a28450e7f7311fba0d'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:38.052775 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1264/2558] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:47:37 2022] DELETE /identity/v3/users/a150dc5df85048a28450e7f7311fba0d => generated 0 bytes in 530 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:38.058318 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8527dba6-d94e-4cea-bd49-0b21ac23d097 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:38.059076 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8527dba6-d94e-4cea-bd49-0b21ac23d097 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:38.059739 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8527dba6-d94e-4cea-bd49-0b21ac23d097 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:38.082410 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-8527dba6-d94e-4cea-bd49-0b21ac23d097 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:38.083681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-8527dba6-d94e-4cea-bd49-0b21ac23d097 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:38.153071 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:DELETE=6,keystone:SELECT=91,keystone:INSERT=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:38.173446 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1295/2559] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:47:38.184803 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a353576-f5bb-4988-a357-653b81797667 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:38.186775 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-3a353576-f5bb-4988-a357-653b81797667 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:38.261526 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a353576-f5bb-4988-a357-653b81797667 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:38.265312 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc6a08e-6967-48b2-a117-c7d29470f0cb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:38.265978 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc6a08e-6967-48b2-a117-c7d29470f0cb service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:38.266266 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc6a08e-6967-48b2-a117-c7d29470f0cb service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:38.266588 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc6a08e-6967-48b2-a117-c7d29470f0cb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:38.276279 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc6a08e-6967-48b2-a117-c7d29470f0cb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:38.278362 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc6a08e-6967-48b2-a117-c7d29470f0cb service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:38.289515 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1265/2560] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:47:38 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:47:38.861436 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8527dba6-d94e-4cea-bd49-0b21ac23d097 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:38.863125 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-8527dba6-d94e-4cea-bd49-0b21ac23d097 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:38.878172 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8527dba6-d94e-4cea-bd49-0b21ac23d097 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:38.884316 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6807dd5b-ba40-412e-a345-3871b3120628 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:38.885535 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6807dd5b-ba40-412e-a345-3871b3120628 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:38.885535 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6807dd5b-ba40-412e-a345-3871b3120628 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:38.885748 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6807dd5b-ba40-412e-a345-3871b3120628 admin admin] PATH_INFO: `/v3/projects/06574e6e01a24509affb68857311b6f1` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:38.887103 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6807dd5b-ba40-412e-a345-3871b3120628 admin admin] RBAC: Authorizing `identity:delete_project(project_id=06574e6e01a24509affb68857311b6f1)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:38.889019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6807dd5b-ba40-412e-a345-3871b3120628 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:39.033157 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-6807dd5b-ba40-412e-a345-3871b3120628 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '06574e6e01a24509affb68857311b6f1', 'request_id': 'req-6807dd5b-ba40-412e-a345-3871b3120628'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:39.043080 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-6807dd5b-ba40-412e-a345-3871b3120628 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '06574e6e01a24509affb68857311b6f1', 'request_id': 'req-6807dd5b-ba40-412e-a345-3871b3120628'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:39.115654 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-6807dd5b-ba40-412e-a345-3871b3120628 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '06574e6e01a24509affb68857311b6f1', 'request_id': 'req-6807dd5b-ba40-412e-a345-3871b3120628'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:39.129613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-6807dd5b-ba40-412e-a345-3871b3120628 admin admin] The token cache is being invalidate because project 06574e6e01a24509affb68857311b6f1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:47:39.129912 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-6807dd5b-ba40-412e-a345-3871b3120628 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:39.130789 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-6807dd5b-ba40-412e-a345-3871b3120628 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:39.132414 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1296/2561] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:47:38 2022] DELETE /identity/v3/projects/06574e6e01a24509affb68857311b6f1 => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:39.137543 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cca8a01-e15f-449f-8326-77b4ab7bcb85 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:39.138036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cca8a01-e15f-449f-8326-77b4ab7bcb85 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:39.138150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cca8a01-e15f-449f-8326-77b4ab7bcb85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:39.144967 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:39.157515 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-8cca8a01-e15f-449f-8326-77b4ab7bcb85 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:39.158807 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-8cca8a01-e15f-449f-8326-77b4ab7bcb85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:39.211123 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6807dd5b-ba40-412e-a345-3871b3120628 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:39.212822 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-6807dd5b-ba40-412e-a345-3871b3120628 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:39.231610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1266/2562] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:47:39.240164 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df03505-bec7-4cac-8533-c60ba9d04f0e None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:39.244916 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df03505-bec7-4cac-8533-c60ba9d04f0e None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:39.244916 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df03505-bec7-4cac-8533-c60ba9d04f0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:39.261861 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-1df03505-bec7-4cac-8533-c60ba9d04f0e None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:39.263639 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-1df03505-bec7-4cac-8533-c60ba9d04f0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:39.277301 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1267/2563] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:47:39.278682 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df03505-bec7-4cac-8533-c60ba9d04f0e None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:39.284296 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6807dd5b-ba40-412e-a345-3871b3120628 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:39.288626 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a4731c-42cc-42fb-a9f1-6060f956dd7b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:39.288789 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df03505-bec7-4cac-8533-c60ba9d04f0e None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:39.289476 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a4731c-42cc-42fb-a9f1-6060f956dd7b service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:39.289871 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a4731c-42cc-42fb-a9f1-6060f956dd7b service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:39.290259 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a4731c-42cc-42fb-a9f1-6060f956dd7b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:39.294039 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-54a4731c-42cc-42fb-a9f1-6060f956dd7b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:39.294958 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93dd6fa6-7512-46a2-b759-e73462f8bc10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:39.295657 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93dd6fa6-7512-46a2-b759-e73462f8bc10 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:39.295925 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93dd6fa6-7512-46a2-b759-e73462f8bc10 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:39.296212 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93dd6fa6-7512-46a2-b759-e73462f8bc10 admin admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:39.296953 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93dd6fa6-7512-46a2-b759-e73462f8bc10 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:47:39.297267 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93dd6fa6-7512-46a2-b759-e73462f8bc10 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:39.298926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93dd6fa6-7512-46a2-b759-e73462f8bc10 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:39.311503 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1268/2564] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:47:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:47:39.315999 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93dd6fa6-7512-46a2-b759-e73462f8bc10 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:39.326132 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93dd6fa6-7512-46a2-b759-e73462f8bc10 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:39.331596 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269ae73d-e668-4d27-880f-cb81774da56c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:39.332256 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269ae73d-e668-4d27-880f-cb81774da56c admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:39.332545 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269ae73d-e668-4d27-880f-cb81774da56c admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:39.332832 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-269ae73d-e668-4d27-880f-cb81774da56c admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:39.333538 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269ae73d-e668-4d27-880f-cb81774da56c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:39.335510 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-269ae73d-e668-4d27-880f-cb81774da56c admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:39.357069 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a4731c-42cc-42fb-a9f1-6060f956dd7b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:39.362910 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a4731c-42cc-42fb-a9f1-6060f956dd7b service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:39.367701 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1297/2565] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:47:39 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:47:39.436154 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1269/2566] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:47:39 2022] POST /identity/v3/projects => generated 403 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:47:39.437664 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a4731c-42cc-42fb-a9f1-6060f956dd7b service cinder] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:39.449336 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a4731c-42cc-42fb-a9f1-6060f956dd7b service cinder] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:39.463031 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:39.509657 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a847aefb-78d8-423f-bfd5-67b39be86006 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:39.510317 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a847aefb-78d8-423f-bfd5-67b39be86006 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:39.510639 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a847aefb-78d8-423f-bfd5-67b39be86006 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:39.510957 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a847aefb-78d8-423f-bfd5-67b39be86006 admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:39.511676 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a847aefb-78d8-423f-bfd5-67b39be86006 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:39.513575 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a847aefb-78d8-423f-bfd5-67b39be86006 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:39.626607 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1298/2567] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:47:39 2022] POST /identity/v3/users => generated 302 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:47:39.627849 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269ae73d-e668-4d27-880f-cb81774da56c admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:39.639251 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-269ae73d-e668-4d27-880f-cb81774da56c admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:39.646597 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47bfcf5d-a538-4e17-91e1-2fca9e5f1274 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:39.646597 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bfcf5d-a538-4e17-91e1-2fca9e5f1274 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:39.646597 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bfcf5d-a538-4e17-91e1-2fca9e5f1274 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:39.646597 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bfcf5d-a538-4e17-91e1-2fca9e5f1274 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:39.647061 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47bfcf5d-a538-4e17-91e1-2fca9e5f1274 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:39.648405 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47bfcf5d-a538-4e17-91e1-2fca9e5f1274 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:39.660484 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1270/2568] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:47:39 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:47:39.665023 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a847aefb-78d8-423f-bfd5-67b39be86006 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:39.677666 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a847aefb-78d8-423f-bfd5-67b39be86006 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:39.683200 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4252cab5-1a61-43d7-a2bb-61ea5103d357 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:39.683994 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4252cab5-1a61-43d7-a2bb-61ea5103d357 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:39.684289 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4252cab5-1a61-43d7-a2bb-61ea5103d357 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:39.684695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4252cab5-1a61-43d7-a2bb-61ea5103d357 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:39.685438 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4252cab5-1a61-43d7-a2bb-61ea5103d357 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:39.687081 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4252cab5-1a61-43d7-a2bb-61ea5103d357 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:39.702224 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1299/2569] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:47:39 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:47:39.703816 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47bfcf5d-a538-4e17-91e1-2fca9e5f1274 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:39.722161 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47bfcf5d-a538-4e17-91e1-2fca9e5f1274 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:39.722161 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-916befb7-6254-480d-881c-7e601f11a9b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:39.722161 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916befb7-6254-480d-881c-7e601f11a9b7 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:39.722593 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916befb7-6254-480d-881c-7e601f11a9b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:39.722593 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916befb7-6254-480d-881c-7e601f11a9b7 admin admin] PATH_INFO: `/v3/projects/94598919aa274c3285568e07dacfe1e5/users/24a35abfd3a6452baca7586a76c0b8d4/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:39.738168 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-916befb7-6254-480d-881c-7e601f11a9b7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=94598919aa274c3285568e07dacfe1e5, user_id=24a35abfd3a6452baca7586a76c0b8d4, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:39.741165 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-916befb7-6254-480d-881c-7e601f11a9b7 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:39.820292 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1271/2570] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:47:39 2022] PUT /identity/v3/projects/94598919aa274c3285568e07dacfe1e5/users/24a35abfd3a6452baca7586a76c0b8d4/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:39.832688 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8bad03-43f4-474d-a362-9d559645098f None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:39.832990 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8bad03-43f4-474d-a362-9d559645098f None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:39.833359 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8bad03-43f4-474d-a362-9d559645098f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:39.861096 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-ff8bad03-43f4-474d-a362-9d559645098f None None] MFA Rules not processed for user `24a35abfd3a6452baca7586a76c0b8d4`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:39.862340 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-ff8bad03-43f4-474d-a362-9d559645098f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:39.929426 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1300/2571] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:39 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:47:39.934151 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-916befb7-6254-480d-881c-7e601f11a9b7 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:39.996233 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-916befb7-6254-480d-881c-7e601f11a9b7 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:40.000979 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b27a985-71e9-4d04-9f1a-50bc0d66aac4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:40.001636 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b27a985-71e9-4d04-9f1a-50bc0d66aac4 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:40.001924 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b27a985-71e9-4d04-9f1a-50bc0d66aac4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:40.002202 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b27a985-71e9-4d04-9f1a-50bc0d66aac4 admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:40.002939 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b27a985-71e9-4d04-9f1a-50bc0d66aac4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:40.004833 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b27a985-71e9-4d04-9f1a-50bc0d66aac4 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:40.101085 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1272/2572] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:47:39 2022] POST /identity/v3/projects => generated 405 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:47:40.105925 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8bad03-43f4-474d-a362-9d559645098f None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:40.116903 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8bad03-43f4-474d-a362-9d559645098f None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:40.176085 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ad8908-4979-4455-b146-bf3dc4159c11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:40.176770 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ad8908-4979-4455-b146-bf3dc4159c11 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:40.177061 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ad8908-4979-4455-b146-bf3dc4159c11 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:40.177343 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ad8908-4979-4455-b146-bf3dc4159c11 admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:40.178046 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ad8908-4979-4455-b146-bf3dc4159c11 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:40.180004 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ad8908-4979-4455-b146-bf3dc4159c11 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:40.282857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1301/2573] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:47:40 2022] POST /identity/v3/users => generated 309 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:47:40.287129 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b27a985-71e9-4d04-9f1a-50bc0d66aac4 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:40.298636 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b27a985-71e9-4d04-9f1a-50bc0d66aac4 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:40.303398 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a42fd44-1ee3-4fe6-82ad-6a4725505700 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:40.304077 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a42fd44-1ee3-4fe6-82ad-6a4725505700 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:40.304370 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a42fd44-1ee3-4fe6-82ad-6a4725505700 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:40.304655 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a42fd44-1ee3-4fe6-82ad-6a4725505700 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:40.305433 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a42fd44-1ee3-4fe6-82ad-6a4725505700 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:40.307104 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a42fd44-1ee3-4fe6-82ad-6a4725505700 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:40.315976 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:40.323637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1273/2574] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:47:40 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:47:40.327679 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ad8908-4979-4455-b146-bf3dc4159c11 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:40.339691 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ad8908-4979-4455-b146-bf3dc4159c11 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:40.345586 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f61290-9abf-4bc1-b60d-546c076456ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:40.345752 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f61290-9abf-4bc1-b60d-546c076456ef admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:40.345968 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f61290-9abf-4bc1-b60d-546c076456ef admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:40.346249 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f61290-9abf-4bc1-b60d-546c076456ef admin admin] PATH_INFO: `/v3/projects/ba984048cb7b4e3b83991919696cb60c/users/732ca0175bf64987bd517f79a6c01890/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:40.356553 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:40.363738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23f61290-9abf-4bc1-b60d-546c076456ef admin admin] RBAC: Authorizing `identity:create_grant(project_id=ba984048cb7b4e3b83991919696cb60c, user_id=732ca0175bf64987bd517f79a6c01890, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:40.367311 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23f61290-9abf-4bc1-b60d-546c076456ef admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:40.431067 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1302/2575] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:47:40 2022] PUT /identity/v3/projects/ba984048cb7b4e3b83991919696cb60c/users/732ca0175bf64987bd517f79a6c01890/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:47:40.435105 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a42fd44-1ee3-4fe6-82ad-6a4725505700 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:40.445803 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a42fd44-1ee3-4fe6-82ad-6a4725505700 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:40.503371 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c7ce9f-e9e2-4e2b-8fc3-af235b61a62a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:40.504062 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c7ce9f-e9e2-4e2b-8fc3-af235b61a62a admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:40.504341 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c7ce9f-e9e2-4e2b-8fc3-af235b61a62a admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:40.504624 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c7ce9f-e9e2-4e2b-8fc3-af235b61a62a admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:40.505430 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c7ce9f-e9e2-4e2b-8fc3-af235b61a62a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:40.507031 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c7ce9f-e9e2-4e2b-8fc3-af235b61a62a admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:40.520528 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1274/2576] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:47:40 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:47:40.521819 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f61290-9abf-4bc1-b60d-546c076456ef admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:40.533259 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f61290-9abf-4bc1-b60d-546c076456ef admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:40.537897 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc9cf71-fbb4-400e-a2b0-86ecdf391a04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:40.538822 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc9cf71-fbb4-400e-a2b0-86ecdf391a04 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:40.539131 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc9cf71-fbb4-400e-a2b0-86ecdf391a04 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:40.539410 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc9cf71-fbb4-400e-a2b0-86ecdf391a04 admin admin] PATH_INFO: `/v3/projects/ba984048cb7b4e3b83991919696cb60c/users/732ca0175bf64987bd517f79a6c01890/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:40.541459 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc9cf71-fbb4-400e-a2b0-86ecdf391a04 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ba984048cb7b4e3b83991919696cb60c, user_id=732ca0175bf64987bd517f79a6c01890, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:40.544867 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cc9cf71-fbb4-400e-a2b0-86ecdf391a04 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:40.555426 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1303/2577] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:47:40 2022] PUT /identity/v3/projects/ba984048cb7b4e3b83991919696cb60c/users/732ca0175bf64987bd517f79a6c01890/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:40.567953 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8d6d5b-8928-4eb3-b0d6-2b891687ef42 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:40.568659 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8d6d5b-8928-4eb3-b0d6-2b891687ef42 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:40.568950 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8d6d5b-8928-4eb3-b0d6-2b891687ef42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:40.596638 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-2a8d6d5b-8928-4eb3-b0d6-2b891687ef42 None None] MFA Rules not processed for user `732ca0175bf64987bd517f79a6c01890`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:40.597850 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-2a8d6d5b-8928-4eb3-b0d6-2b891687ef42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:40.666533 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1275/2578] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:40 2022] POST /identity/v3/auth/tokens => generated 3339 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:47:40.678107 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc9cf71-fbb4-400e-a2b0-86ecdf391a04 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:40.680035 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-9cc9cf71-fbb4-400e-a2b0-86ecdf391a04 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:40.739640 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc9cf71-fbb4-400e-a2b0-86ecdf391a04 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:40.743343 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5ef875-c9a4-45e4-8968-48ea69529d05 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:40.743986 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5ef875-c9a4-45e4-8968-48ea69529d05 service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:40.744275 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5ef875-c9a4-45e4-8968-48ea69529d05 service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:40.744556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5ef875-c9a4-45e4-8968-48ea69529d05 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:40.754838 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e5ef875-c9a4-45e4-8968-48ea69529d05 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:40.762863 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e5ef875-c9a4-45e4-8968-48ea69529d05 service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:40.769301 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1304/2579] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:47:40 2022] GET /identity/v3/auth/tokens => generated 3339 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:47:40.933761 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8d6d5b-8928-4eb3-b0d6-2b891687ef42 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:41.009500 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8d6d5b-8928-4eb3-b0d6-2b891687ef42 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:41.014020 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7311b45-8170-4121-b183-836cb9fa8ae8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:41.014920 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7311b45-8170-4121-b183-836cb9fa8ae8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:41.015209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7311b45-8170-4121-b183-836cb9fa8ae8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:41.015487 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7311b45-8170-4121-b183-836cb9fa8ae8 admin admin] PATH_INFO: `/v3/users/24a35abfd3a6452baca7586a76c0b8d4` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:41.016675 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7311b45-8170-4121-b183-836cb9fa8ae8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=24a35abfd3a6452baca7586a76c0b8d4)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:41.018615 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7311b45-8170-4121-b183-836cb9fa8ae8 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:41.043977 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:41.205962 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-c7311b45-8170-4121-b183-836cb9fa8ae8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '24a35abfd3a6452baca7586a76c0b8d4'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:41.275948 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-c7311b45-8170-4121-b183-836cb9fa8ae8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '24a35abfd3a6452baca7586a76c0b8d4'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:41.277187 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-c7311b45-8170-4121-b183-836cb9fa8ae8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '24a35abfd3a6452baca7586a76c0b8d4'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:41.284820 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-c7311b45-8170-4121-b183-836cb9fa8ae8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '24a35abfd3a6452baca7586a76c0b8d4'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:41.300443 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-c7311b45-8170-4121-b183-836cb9fa8ae8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '24a35abfd3a6452baca7586a76c0b8d4'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:41.303938 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1276/2580] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:47:40 2022] DELETE /identity/v3/users/24a35abfd3a6452baca7586a76c0b8d4 => generated 0 bytes in 371 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:41.308553 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706fa563-3720-45d7-ba27-b03abcf7b9ce None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:41.308998 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706fa563-3720-45d7-ba27-b03abcf7b9ce None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:41.309104 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706fa563-3720-45d7-ba27-b03abcf7b9ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:41.326677 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-706fa563-3720-45d7-ba27-b03abcf7b9ce None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:41.329206 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-706fa563-3720-45d7-ba27-b03abcf7b9ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:41.388559 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1305/2581] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:47:41.398351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7311b45-8170-4121-b183-836cb9fa8ae8 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:41.399838 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-c7311b45-8170-4121-b183-836cb9fa8ae8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:41.460751 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7311b45-8170-4121-b183-836cb9fa8ae8 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:41.465043 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6828949-4d65-4d44-be40-904eb60b94fb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:41.465687 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6828949-4d65-4d44-be40-904eb60b94fb service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:41.465969 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6828949-4d65-4d44-be40-904eb60b94fb service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:41.466245 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6828949-4d65-4d44-be40-904eb60b94fb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:41.475680 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6828949-4d65-4d44-be40-904eb60b94fb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:41.477111 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6828949-4d65-4d44-be40-904eb60b94fb service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:41.487372 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1277/2582] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:47:41 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:47:41.912756 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706fa563-3720-45d7-ba27-b03abcf7b9ce None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:41.914367 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-706fa563-3720-45d7-ba27-b03abcf7b9ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:41.927817 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706fa563-3720-45d7-ba27-b03abcf7b9ce None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:41.932556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd768024-170d-42d5-9791-76298ced0556 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:41.934124 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd768024-170d-42d5-9791-76298ced0556 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:41.940088 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd768024-170d-42d5-9791-76298ced0556 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:41.940088 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd768024-170d-42d5-9791-76298ced0556 admin admin] PATH_INFO: `/v3/projects/94598919aa274c3285568e07dacfe1e5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:41.940088 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd768024-170d-42d5-9791-76298ced0556 admin admin] RBAC: Authorizing `identity:delete_project(project_id=94598919aa274c3285568e07dacfe1e5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:41.940088 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd768024-170d-42d5-9791-76298ced0556 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:42.093046 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-bd768024-170d-42d5-9791-76298ced0556 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '94598919aa274c3285568e07dacfe1e5', 'request_id': 'req-bd768024-170d-42d5-9791-76298ced0556'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:42.099557 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-bd768024-170d-42d5-9791-76298ced0556 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '94598919aa274c3285568e07dacfe1e5', 'request_id': 'req-bd768024-170d-42d5-9791-76298ced0556'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:42.212323 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-bd768024-170d-42d5-9791-76298ced0556 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '94598919aa274c3285568e07dacfe1e5', 'request_id': 'req-bd768024-170d-42d5-9791-76298ced0556'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:42.221961 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:42.225975 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-bd768024-170d-42d5-9791-76298ced0556 admin admin] The token cache is being invalidate because project 94598919aa274c3285568e07dacfe1e5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:47:42.226273 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-bd768024-170d-42d5-9791-76298ced0556 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:42.226959 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-bd768024-170d-42d5-9791-76298ced0556 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:42.228569 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1306/2583] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:47:41 2022] DELETE /identity/v3/projects/94598919aa274c3285568e07dacfe1e5 => generated 0 bytes in 316 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:42.233045 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6828949-4d65-4d44-be40-904eb60b94fb service neutron] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:42.242863 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6828949-4d65-4d44-be40-904eb60b94fb service neutron] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:42.305022 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c2ce44-cbdd-41f1-9913-60433d612ac7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:42.306686 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c2ce44-cbdd-41f1-9913-60433d612ac7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:42.306981 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c2ce44-cbdd-41f1-9913-60433d612ac7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:42.307267 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c2ce44-cbdd-41f1-9913-60433d612ac7 admin admin] PATH_INFO: `/v3/users/732ca0175bf64987bd517f79a6c01890` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:42.309648 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c2ce44-cbdd-41f1-9913-60433d612ac7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=732ca0175bf64987bd517f79a6c01890)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:42.311669 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c2ce44-cbdd-41f1-9913-60433d612ac7 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:42.496043 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:42.498083 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-b0c2ce44-cbdd-41f1-9913-60433d612ac7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '732ca0175bf64987bd517f79a6c01890'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:42.562330 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-b0c2ce44-cbdd-41f1-9913-60433d612ac7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '732ca0175bf64987bd517f79a6c01890'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:42.563127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-b0c2ce44-cbdd-41f1-9913-60433d612ac7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '732ca0175bf64987bd517f79a6c01890'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:42.570995 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-b0c2ce44-cbdd-41f1-9913-60433d612ac7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '732ca0175bf64987bd517f79a6c01890'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:42.584081 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-b0c2ce44-cbdd-41f1-9913-60433d612ac7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '732ca0175bf64987bd517f79a6c01890'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:42.590576 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1278/2584] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:47:42 2022] DELETE /identity/v3/users/732ca0175bf64987bd517f79a6c01890 => generated 0 bytes in 358 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:43.033608 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd768024-170d-42d5-9791-76298ced0556 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:43.035138 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-bd768024-170d-42d5-9791-76298ced0556 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:43.054279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd768024-170d-42d5-9791-76298ced0556 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:43.113576 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5304127-96c2-47c3-9e26-84d7806936fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:43.114466 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5304127-96c2-47c3-9e26-84d7806936fd admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:43.114818 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5304127-96c2-47c3-9e26-84d7806936fd admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:43.115094 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5304127-96c2-47c3-9e26-84d7806936fd admin admin] PATH_INFO: `/v3/projects/ba984048cb7b4e3b83991919696cb60c` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:43.116281 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5304127-96c2-47c3-9e26-84d7806936fd admin admin] RBAC: Authorizing `identity:delete_project(project_id=ba984048cb7b4e3b83991919696cb60c)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:43.118180 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5304127-96c2-47c3-9e26-84d7806936fd admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:43.278846 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e5304127-96c2-47c3-9e26-84d7806936fd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ba984048cb7b4e3b83991919696cb60c', 'request_id': 'req-e5304127-96c2-47c3-9e26-84d7806936fd'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:43.291122 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e5304127-96c2-47c3-9e26-84d7806936fd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ba984048cb7b4e3b83991919696cb60c', 'request_id': 'req-e5304127-96c2-47c3-9e26-84d7806936fd'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:43.367484 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e5304127-96c2-47c3-9e26-84d7806936fd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ba984048cb7b4e3b83991919696cb60c', 'request_id': 'req-e5304127-96c2-47c3-9e26-84d7806936fd'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:43.379227 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e5304127-96c2-47c3-9e26-84d7806936fd admin admin] The token cache is being invalidate because project ba984048cb7b4e3b83991919696cb60c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:47:43.379520 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e5304127-96c2-47c3-9e26-84d7806936fd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:43.380159 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e5304127-96c2-47c3-9e26-84d7806936fd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:43.381705 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1307/2585] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:47:43 2022] DELETE /identity/v3/projects/ba984048cb7b4e3b83991919696cb60c => generated 0 bytes in 349 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:43.386935 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cdb278-2e48-4bb3-bb65-2af261157b40 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:43.387437 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cdb278-2e48-4bb3-bb65-2af261157b40 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:43.387545 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cdb278-2e48-4bb3-bb65-2af261157b40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:43.406892 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-b6cdb278-2e48-4bb3-bb65-2af261157b40 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:43.408173 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-b6cdb278-2e48-4bb3-bb65-2af261157b40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:43.478230 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1279/2586] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:43 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:47:43.486257 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665f5e69-f549-497a-8568-ab176baea467 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:43.486603 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665f5e69-f549-497a-8568-ab176baea467 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:43.486885 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665f5e69-f549-497a-8568-ab176baea467 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:43.499736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6cdb278-2e48-4bb3-bb65-2af261157b40 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:43.501239 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-b6cdb278-2e48-4bb3-bb65-2af261157b40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:43.507038 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-665f5e69-f549-497a-8568-ab176baea467 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:43.508571 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-665f5e69-f549-497a-8568-ab176baea467 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:43.514262 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6cdb278-2e48-4bb3-bb65-2af261157b40 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:43.518629 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1308/2587] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:43 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:47:43.520478 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c5263a-4b09-48df-890e-7201f2e994b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:43.525438 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c5263a-4b09-48df-890e-7201f2e994b5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:43.525438 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c5263a-4b09-48df-890e-7201f2e994b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:43.525438 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c5263a-4b09-48df-890e-7201f2e994b5 admin admin] PATH_INFO: `/v3/users/43f0d4a573ba4fd59666d1b131d62779` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:43.525438 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c5263a-4b09-48df-890e-7201f2e994b5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=43f0d4a573ba4fd59666d1b131d62779)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:43.525438 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c5263a-4b09-48df-890e-7201f2e994b5 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:43.529364 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665f5e69-f549-497a-8568-ab176baea467 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:43.539027 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665f5e69-f549-497a-8568-ab176baea467 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:43.544269 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb44421d-b177-4889-b2c7-1da58d6dcc1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:43.544927 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb44421d-b177-4889-b2c7-1da58d6dcc1d admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:43.545214 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb44421d-b177-4889-b2c7-1da58d6dcc1d admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:43.545651 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb44421d-b177-4889-b2c7-1da58d6dcc1d admin admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:43.546382 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb44421d-b177-4889-b2c7-1da58d6dcc1d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:47:43.546870 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb44421d-b177-4889-b2c7-1da58d6dcc1d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:43.548611 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb44421d-b177-4889-b2c7-1da58d6dcc1d admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:43.561162 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1309/2588] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:47:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:47:43.565515 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb44421d-b177-4889-b2c7-1da58d6dcc1d admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:43.576912 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb44421d-b177-4889-b2c7-1da58d6dcc1d admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:43.582349 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9842b388-4b81-402f-8e5a-b0e9827872b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:43.583204 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9842b388-4b81-402f-8e5a-b0e9827872b6 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:43.583446 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9842b388-4b81-402f-8e5a-b0e9827872b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:43.583728 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9842b388-4b81-402f-8e5a-b0e9827872b6 admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:43.584426 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9842b388-4b81-402f-8e5a-b0e9827872b6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:43.586331 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9842b388-4b81-402f-8e5a-b0e9827872b6 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:43.718549 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1310/2589] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:47:43 2022] POST /identity/v3/projects => generated 409 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:47:43.722601 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9842b388-4b81-402f-8e5a-b0e9827872b6 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:43.771940 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:43.789346 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9842b388-4b81-402f-8e5a-b0e9827872b6 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:43.790881 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0239be49-1b05-4767-9f43-28664a18e2bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:43.791574 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0239be49-1b05-4767-9f43-28664a18e2bb admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:43.791845 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0239be49-1b05-4767-9f43-28664a18e2bb admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:43.792203 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0239be49-1b05-4767-9f43-28664a18e2bb admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:43.793182 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0239be49-1b05-4767-9f43-28664a18e2bb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:43.795308 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0239be49-1b05-4767-9f43-28664a18e2bb admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:43.828265 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-26c5263a-4b09-48df-890e-7201f2e994b5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '43f0d4a573ba4fd59666d1b131d62779'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:43.891261 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1311/2590] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:47:43 2022] POST /identity/v3/users => generated 305 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:47:43.892576 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0239be49-1b05-4767-9f43-28664a18e2bb admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:43.902903 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0239be49-1b05-4767-9f43-28664a18e2bb admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:43.907695 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40be3726-129f-4abe-8a42-fb22ed84e678 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:43.908365 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40be3726-129f-4abe-8a42-fb22ed84e678 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:43.908652 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40be3726-129f-4abe-8a42-fb22ed84e678 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:43.908933 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40be3726-129f-4abe-8a42-fb22ed84e678 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:43.909670 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40be3726-129f-4abe-8a42-fb22ed84e678 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:43.911295 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40be3726-129f-4abe-8a42-fb22ed84e678 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:43.915025 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-26c5263a-4b09-48df-890e-7201f2e994b5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '43f0d4a573ba4fd59666d1b131d62779'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:43.915694 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-26c5263a-4b09-48df-890e-7201f2e994b5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '43f0d4a573ba4fd59666d1b131d62779'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:43.923619 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1312/2591] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:47:43 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 12 17:47:43.925039 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-26c5263a-4b09-48df-890e-7201f2e994b5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '43f0d4a573ba4fd59666d1b131d62779'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:43.928373 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40be3726-129f-4abe-8a42-fb22ed84e678 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:43.930110 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-40be3726-129f-4abe-8a42-fb22ed84e678 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:43.939243 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-26c5263a-4b09-48df-890e-7201f2e994b5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '43f0d4a573ba4fd59666d1b131d62779'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:43.944213 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40be3726-129f-4abe-8a42-fb22ed84e678 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:43.946138 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1280/2592] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:47:43 2022] DELETE /identity/v3/users/43f0d4a573ba4fd59666d1b131d62779 => generated 0 bytes in 447 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:47:43.951345 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35754206-28d4-4267-9e59-6ca53c4f6e7c None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:43.951661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35754206-28d4-4267-9e59-6ca53c4f6e7c None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:43.951938 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35754206-28d4-4267-9e59-6ca53c4f6e7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:43.970270 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-35754206-28d4-4267-9e59-6ca53c4f6e7c None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:43.971754 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-35754206-28d4-4267-9e59-6ca53c4f6e7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:44.004361 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=85,keystone:INSERT=2,keystone:DELETE=13 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:44.007575 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0670e88-d863-4e9c-bf1d-263c6416b6b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:44.008250 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0670e88-d863-4e9c-bf1d-263c6416b6b7 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:44.008559 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0670e88-d863-4e9c-bf1d-263c6416b6b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:44.008812 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0670e88-d863-4e9c-bf1d-263c6416b6b7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:44.009542 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0670e88-d863-4e9c-bf1d-263c6416b6b7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:44.011204 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0670e88-d863-4e9c-bf1d-263c6416b6b7 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:44.026084 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1313/2593] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:47:43 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 12 17:47:44.027011 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0670e88-d863-4e9c-bf1d-263c6416b6b7 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:44.035528 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1281/2594] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:43 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:47:44.038528 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0670e88-d863-4e9c-bf1d-263c6416b6b7 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:44.044235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5505922-9272-477f-8323-c9447b254648 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:44.045070 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5505922-9272-477f-8323-c9447b254648 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:44.045358 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5505922-9272-477f-8323-c9447b254648 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:44.045719 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5505922-9272-477f-8323-c9447b254648 admin admin] PATH_INFO: `/v3/projects/682341e3a9254b3e82adc1bf50a7462d/users/e4413ac8b045432cb9ea8a156fa12571/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:44.046815 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35754206-28d4-4267-9e59-6ca53c4f6e7c None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:44.048368 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-35754206-28d4-4267-9e59-6ca53c4f6e7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:44.062533 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5505922-9272-477f-8323-c9447b254648 admin admin] RBAC: Authorizing `identity:create_grant(project_id=682341e3a9254b3e82adc1bf50a7462d, user_id=e4413ac8b045432cb9ea8a156fa12571, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:44.065260 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5505922-9272-477f-8323-c9447b254648 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:44.108920 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35754206-28d4-4267-9e59-6ca53c4f6e7c None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:44.112796 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-312900e8-c902-4c86-ab1c-a302b9151876 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:44.113452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312900e8-c902-4c86-ab1c-a302b9151876 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:44.113746 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312900e8-c902-4c86-ab1c-a302b9151876 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:44.114039 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312900e8-c902-4c86-ab1c-a302b9151876 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:44.123987 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-312900e8-c902-4c86-ab1c-a302b9151876 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:44.126577 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-312900e8-c902-4c86-ab1c-a302b9151876 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:44.135511 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1282/2595] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:47:44 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:47:44.185840 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1314/2596] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:47:44 2022] PUT /identity/v3/projects/682341e3a9254b3e82adc1bf50a7462d/users/e4413ac8b045432cb9ea8a156fa12571/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:44.198344 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49eaaba4-4f90-4961-b365-110960f8ee8b None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:44.198702 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49eaaba4-4f90-4961-b365-110960f8ee8b None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:44.199039 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49eaaba4-4f90-4961-b365-110960f8ee8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:44.228320 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-49eaaba4-4f90-4961-b365-110960f8ee8b None None] MFA Rules not processed for user `e4413ac8b045432cb9ea8a156fa12571`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:44.230024 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-49eaaba4-4f90-4961-b365-110960f8ee8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:44.297558 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1283/2597] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:44 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:47:44.301791 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5505922-9272-477f-8323-c9447b254648 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:44.383352 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5505922-9272-477f-8323-c9447b254648 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:44.391140 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda4c7df-a87a-4d53-b34b-fe9bb00fe94d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:44.391825 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda4c7df-a87a-4d53-b34b-fe9bb00fe94d admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:44.392117 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda4c7df-a87a-4d53-b34b-fe9bb00fe94d admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:44.392401 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda4c7df-a87a-4d53-b34b-fe9bb00fe94d admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:44.393135 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda4c7df-a87a-4d53-b34b-fe9bb00fe94d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:44.395113 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda4c7df-a87a-4d53-b34b-fe9bb00fe94d admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:44.520903 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1315/2598] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:47:44 2022] POST /identity/v3/projects => generated 407 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:47:44.521987 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49eaaba4-4f90-4961-b365-110960f8ee8b None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:44.533389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49eaaba4-4f90-4961-b365-110960f8ee8b None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:44.602990 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7d9c96-bbcd-49e1-a222-8a62797c9a9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:44.603677 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7d9c96-bbcd-49e1-a222-8a62797c9a9d admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:44.604271 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7d9c96-bbcd-49e1-a222-8a62797c9a9d admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:44.604372 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7d9c96-bbcd-49e1-a222-8a62797c9a9d admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:44.604997 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f7d9c96-bbcd-49e1-a222-8a62797c9a9d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:44.607020 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f7d9c96-bbcd-49e1-a222-8a62797c9a9d admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:44.661784 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda4c7df-a87a-4d53-b34b-fe9bb00fe94d admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:44.661947 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-eda4c7df-a87a-4d53-b34b-fe9bb00fe94d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:44.680394 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda4c7df-a87a-4d53-b34b-fe9bb00fe94d admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:44.690965 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:44.691498 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ffa8f9-8e06-4d04-b5db-25c1aac178a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:44.691498 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ffa8f9-8e06-4d04-b5db-25c1aac178a9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:44.691498 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ffa8f9-8e06-4d04-b5db-25c1aac178a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:44.691498 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ffa8f9-8e06-4d04-b5db-25c1aac178a9 admin admin] PATH_INFO: `/v3/projects/dfa2e8f4698b4e1bb9beac6c8ba85083` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:44.691865 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ffa8f9-8e06-4d04-b5db-25c1aac178a9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dfa2e8f4698b4e1bb9beac6c8ba85083)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:44.692382 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ffa8f9-8e06-4d04-b5db-25c1aac178a9 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:44.696247 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1284/2599] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:47:44 2022] POST /identity/v3/users => generated 310 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:47:44.701876 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7d9c96-bbcd-49e1-a222-8a62797c9a9d admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:44.712122 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7d9c96-bbcd-49e1-a222-8a62797c9a9d admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:44.716328 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:44.717066 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0803f2-2ab9-45d5-aaca-aa4452328673 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:44.717785 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0803f2-2ab9-45d5-aaca-aa4452328673 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:44.718132 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0803f2-2ab9-45d5-aaca-aa4452328673 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:44.718357 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0803f2-2ab9-45d5-aaca-aa4452328673 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:44.719184 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed0803f2-2ab9-45d5-aaca-aa4452328673 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:44.720764 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed0803f2-2ab9-45d5-aaca-aa4452328673 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:44.731243 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1285/2600] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:47:44 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:47:44.735230 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0803f2-2ab9-45d5-aaca-aa4452328673 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:44.745070 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0803f2-2ab9-45d5-aaca-aa4452328673 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:44.750678 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096f6d76-59a1-4569-9123-5e45b035c3ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:44.751565 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096f6d76-59a1-4569-9123-5e45b035c3ec admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:44.751864 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096f6d76-59a1-4569-9123-5e45b035c3ec admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:44.752150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096f6d76-59a1-4569-9123-5e45b035c3ec admin admin] PATH_INFO: `/v3/projects/2e2557e412f24b92a6cfc6edcfa306d7/users/e357115b18624b00a58524079d297b0c/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:44.769296 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096f6d76-59a1-4569-9123-5e45b035c3ec admin admin] RBAC: Authorizing `identity:create_grant(project_id=2e2557e412f24b92a6cfc6edcfa306d7, user_id=e357115b18624b00a58524079d297b0c, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:44.772174 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096f6d76-59a1-4569-9123-5e45b035c3ec admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:44.870584 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1286/2601] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:47:44 2022] PUT /identity/v3/projects/2e2557e412f24b92a6cfc6edcfa306d7/users/e357115b18624b00a58524079d297b0c/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:47:44.875099 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096f6d76-59a1-4569-9123-5e45b035c3ec admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:44.878819 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-f9ffa8f9-8e06-4d04-b5db-25c1aac178a9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dfa2e8f4698b4e1bb9beac6c8ba85083', 'request_id': 'req-f9ffa8f9-8e06-4d04-b5db-25c1aac178a9'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:44.884207 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-f9ffa8f9-8e06-4d04-b5db-25c1aac178a9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dfa2e8f4698b4e1bb9beac6c8ba85083', 'request_id': 'req-f9ffa8f9-8e06-4d04-b5db-25c1aac178a9'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:44.936052 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096f6d76-59a1-4569-9123-5e45b035c3ec admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:44.973124 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-f9ffa8f9-8e06-4d04-b5db-25c1aac178a9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dfa2e8f4698b4e1bb9beac6c8ba85083', 'request_id': 'req-f9ffa8f9-8e06-4d04-b5db-25c1aac178a9'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:44.987074 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-f9ffa8f9-8e06-4d04-b5db-25c1aac178a9 admin admin] The token cache is being invalidate because project dfa2e8f4698b4e1bb9beac6c8ba85083 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:47:44.987370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-f9ffa8f9-8e06-4d04-b5db-25c1aac178a9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:44.988048 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-f9ffa8f9-8e06-4d04-b5db-25c1aac178a9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:44.989681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1316/2602] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:47:44 2022] DELETE /identity/v3/projects/dfa2e8f4698b4e1bb9beac6c8ba85083 => generated 0 bytes in 330 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:47:44.992809 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5108862a-a157-4e26-9abc-405452837157 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:44.996342 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5108862a-a157-4e26-9abc-405452837157 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:44.996342 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5108862a-a157-4e26-9abc-405452837157 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:44.997085 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ffa8f9-8e06-4d04-b5db-25c1aac178a9 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:44.997194 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5108862a-a157-4e26-9abc-405452837157 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:44.997194 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5108862a-a157-4e26-9abc-405452837157 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:44.997382 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-f9ffa8f9-8e06-4d04-b5db-25c1aac178a9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:44.997491 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5108862a-a157-4e26-9abc-405452837157 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:45.008117 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1287/2603] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:47:44 2022] GET /identity/v3/roles => generated 1420 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 12 17:47:45.009764 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ffa8f9-8e06-4d04-b5db-25c1aac178a9 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:45.012708 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5108862a-a157-4e26-9abc-405452837157 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:45.014578 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-5108862a-a157-4e26-9abc-405452837157 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:45.015113 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a0fadb-1af5-47a0-827c-65fd11cf1098 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:45.016003 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a0fadb-1af5-47a0-827c-65fd11cf1098 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:45.016455 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a0fadb-1af5-47a0-827c-65fd11cf1098 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:45.016743 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a0fadb-1af5-47a0-827c-65fd11cf1098 admin admin] PATH_INFO: `/v3/users/59a43d118c004751989f8eec8d38728c` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:45.018063 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07a0fadb-1af5-47a0-827c-65fd11cf1098 admin admin] RBAC: Authorizing `identity:delete_user(user_id=59a43d118c004751989f8eec8d38728c)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:45.019956 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07a0fadb-1af5-47a0-827c-65fd11cf1098 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:45.030117 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5108862a-a157-4e26-9abc-405452837157 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:45.035172 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b73a85a-bb96-446f-a857-94ae26fee8df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:45.036085 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b73a85a-bb96-446f-a857-94ae26fee8df admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:45.036388 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b73a85a-bb96-446f-a857-94ae26fee8df admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:45.036656 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b73a85a-bb96-446f-a857-94ae26fee8df admin admin] PATH_INFO: `/v3/projects/2e2557e412f24b92a6cfc6edcfa306d7/users/e357115b18624b00a58524079d297b0c/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:45.038694 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b73a85a-bb96-446f-a857-94ae26fee8df admin admin] RBAC: Authorizing `identity:create_grant(project_id=2e2557e412f24b92a6cfc6edcfa306d7, user_id=e357115b18624b00a58524079d297b0c, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:45.041503 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b73a85a-bb96-446f-a857-94ae26fee8df admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:45.053373 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1288/2604] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:47:45 2022] PUT /identity/v3/projects/2e2557e412f24b92a6cfc6edcfa306d7/users/e357115b18624b00a58524079d297b0c/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:47:45.064283 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcdf514-8416-4bbd-a129-fdef4e375168 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:45.064581 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcdf514-8416-4bbd-a129-fdef4e375168 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:45.064867 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcdf514-8416-4bbd-a129-fdef4e375168 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:45.093376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-5bcdf514-8416-4bbd-a129-fdef4e375168 None None] MFA Rules not processed for user `e357115b18624b00a58524079d297b0c`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:45.095272 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-5bcdf514-8416-4bbd-a129-fdef4e375168 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:45.142154 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:45.169259 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1289/2605] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:45 2022] POST /identity/v3/auth/tokens => generated 3341 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:47:45.181784 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bcdf514-8416-4bbd-a129-fdef4e375168 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:45.183362 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-5bcdf514-8416-4bbd-a129-fdef4e375168 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:45.223214 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-07a0fadb-1af5-47a0-827c-65fd11cf1098 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '59a43d118c004751989f8eec8d38728c'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:45.223876 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-07a0fadb-1af5-47a0-827c-65fd11cf1098 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '59a43d118c004751989f8eec8d38728c'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:45.224511 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-07a0fadb-1af5-47a0-827c-65fd11cf1098 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '59a43d118c004751989f8eec8d38728c'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:45.232940 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-07a0fadb-1af5-47a0-827c-65fd11cf1098 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '59a43d118c004751989f8eec8d38728c'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:45.245122 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bcdf514-8416-4bbd-a129-fdef4e375168 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:45.246146 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-07a0fadb-1af5-47a0-827c-65fd11cf1098 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '59a43d118c004751989f8eec8d38728c'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:45.249691 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cdab33e-a996-485b-a8d8-e74c4d2ea67e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:45.250350 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cdab33e-a996-485b-a8d8-e74c4d2ea67e service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:45.250674 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cdab33e-a996-485b-a8d8-e74c4d2ea67e service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:45.250964 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cdab33e-a996-485b-a8d8-e74c4d2ea67e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:45.252878 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-5cdab33e-a996-485b-a8d8-e74c4d2ea67e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:45.265180 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cdab33e-a996-485b-a8d8-e74c4d2ea67e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:45.266666 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cdab33e-a996-485b-a8d8-e74c4d2ea67e service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:45.276992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1290/2606] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:47:45 2022] GET /identity/v3/auth/tokens => generated 3341 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:47:45.335771 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1317/2607] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:47:44 2022] DELETE /identity/v3/users/59a43d118c004751989f8eec8d38728c => generated 0 bytes in 342 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:47:45.660766 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cdab33e-a996-485b-a8d8-e74c4d2ea67e service cinder] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:45.662378 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-5cdab33e-a996-485b-a8d8-e74c4d2ea67e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:45.676860 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cdab33e-a996-485b-a8d8-e74c4d2ea67e service cinder] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:45.744076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe1eb36-7843-4d95-adff-9b02e023b10c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:45.744954 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe1eb36-7843-4d95-adff-9b02e023b10c service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:45.745134 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe1eb36-7843-4d95-adff-9b02e023b10c service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:45.745495 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe1eb36-7843-4d95-adff-9b02e023b10c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:45.747650 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-9fe1eb36-7843-4d95-adff-9b02e023b10c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:45.823834 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe1eb36-7843-4d95-adff-9b02e023b10c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:45.825893 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe1eb36-7843-4d95-adff-9b02e023b10c service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:45.837197 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1291/2608] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:47:45 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 177 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:47:45.900938 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a0fadb-1af5-47a0-827c-65fd11cf1098 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:45.900938 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-07a0fadb-1af5-47a0-827c-65fd11cf1098 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:45.985644 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a0fadb-1af5-47a0-827c-65fd11cf1098 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:45.991490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e58c4f2-095e-4226-8726-12eedaa47897 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:45.992392 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e58c4f2-095e-4226-8726-12eedaa47897 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:45.992676 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e58c4f2-095e-4226-8726-12eedaa47897 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:45.992953 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e58c4f2-095e-4226-8726-12eedaa47897 admin admin] PATH_INFO: `/v3/projects/2ccc5363c96e4eaf96ff870dea7bd647` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:45.994805 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e58c4f2-095e-4226-8726-12eedaa47897 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2ccc5363c96e4eaf96ff870dea7bd647)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:45.997080 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e58c4f2-095e-4226-8726-12eedaa47897 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:46.024195 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:46.136788 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-9e58c4f2-095e-4226-8726-12eedaa47897 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2ccc5363c96e4eaf96ff870dea7bd647', 'request_id': 'req-9e58c4f2-095e-4226-8726-12eedaa47897'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:46.143595 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-9e58c4f2-095e-4226-8726-12eedaa47897 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2ccc5363c96e4eaf96ff870dea7bd647', 'request_id': 'req-9e58c4f2-095e-4226-8726-12eedaa47897'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:46.234751 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-9e58c4f2-095e-4226-8726-12eedaa47897 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2ccc5363c96e4eaf96ff870dea7bd647', 'request_id': 'req-9e58c4f2-095e-4226-8726-12eedaa47897'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:46.247024 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-9e58c4f2-095e-4226-8726-12eedaa47897 admin admin] The token cache is being invalidate because project 2ccc5363c96e4eaf96ff870dea7bd647 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:47:46.247355 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-9e58c4f2-095e-4226-8726-12eedaa47897 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:46.248018 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-9e58c4f2-095e-4226-8726-12eedaa47897 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:46.249560 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1318/2609] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:47:45 2022] DELETE /identity/v3/projects/2ccc5363c96e4eaf96ff870dea7bd647 => generated 0 bytes in 354 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:46.253872 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe1eb36-7843-4d95-adff-9b02e023b10c service cinder] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:46.265886 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe1eb36-7843-4d95-adff-9b02e023b10c service cinder] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:46.299479 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:46.325272 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa29f7a-9233-400a-9163-4208bd0fce85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:46.326165 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa29f7a-9233-400a-9163-4208bd0fce85 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:46.326479 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa29f7a-9233-400a-9163-4208bd0fce85 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:46.326768 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa29f7a-9233-400a-9163-4208bd0fce85 admin admin] PATH_INFO: `/v3/users/e66371b16f464f6faaa0a16157cfd6ee` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:46.328041 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fa29f7a-9233-400a-9163-4208bd0fce85 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e66371b16f464f6faaa0a16157cfd6ee)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:46.329984 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fa29f7a-9233-400a-9163-4208bd0fce85 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:46.514924 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-3fa29f7a-9233-400a-9163-4208bd0fce85 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e66371b16f464f6faaa0a16157cfd6ee'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:46.607325 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-3fa29f7a-9233-400a-9163-4208bd0fce85 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e66371b16f464f6faaa0a16157cfd6ee'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:46.607952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-3fa29f7a-9233-400a-9163-4208bd0fce85 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e66371b16f464f6faaa0a16157cfd6ee'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:46.615792 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-3fa29f7a-9233-400a-9163-4208bd0fce85 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e66371b16f464f6faaa0a16157cfd6ee'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:46.633660 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-3fa29f7a-9233-400a-9163-4208bd0fce85 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e66371b16f464f6faaa0a16157cfd6ee'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:46.641187 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1292/2610] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:47:46 2022] DELETE /identity/v3/users/e66371b16f464f6faaa0a16157cfd6ee => generated 0 bytes in 388 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:47.067204 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e58c4f2-095e-4226-8726-12eedaa47897 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:47.068992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-9e58c4f2-095e-4226-8726-12eedaa47897 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:47.085582 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e58c4f2-095e-4226-8726-12eedaa47897 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:47.150601 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ccccf42-08da-43da-8904-a3d4699be3ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:47.151471 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ccccf42-08da-43da-8904-a3d4699be3ea admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:47.151756 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ccccf42-08da-43da-8904-a3d4699be3ea admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:47.152037 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ccccf42-08da-43da-8904-a3d4699be3ea admin admin] PATH_INFO: `/v3/projects/67749de63a36467f86a5c66292912a32` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:47.153399 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ccccf42-08da-43da-8904-a3d4699be3ea admin admin] RBAC: Authorizing `identity:delete_project(project_id=67749de63a36467f86a5c66292912a32)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:47.155476 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ccccf42-08da-43da-8904-a3d4699be3ea admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:47.324145 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-7ccccf42-08da-43da-8904-a3d4699be3ea admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '67749de63a36467f86a5c66292912a32', 'request_id': 'req-7ccccf42-08da-43da-8904-a3d4699be3ea'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:47.331189 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-7ccccf42-08da-43da-8904-a3d4699be3ea admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '67749de63a36467f86a5c66292912a32', 'request_id': 'req-7ccccf42-08da-43da-8904-a3d4699be3ea'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:47.424705 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-7ccccf42-08da-43da-8904-a3d4699be3ea admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '67749de63a36467f86a5c66292912a32', 'request_id': 'req-7ccccf42-08da-43da-8904-a3d4699be3ea'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:47.438948 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-7ccccf42-08da-43da-8904-a3d4699be3ea admin admin] The token cache is being invalidate because project 67749de63a36467f86a5c66292912a32 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:47:47.439260 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-7ccccf42-08da-43da-8904-a3d4699be3ea admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:47.439920 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-7ccccf42-08da-43da-8904-a3d4699be3ea admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:47.441617 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1319/2611] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:47:47 2022] DELETE /identity/v3/projects/67749de63a36467f86a5c66292912a32 => generated 0 bytes in 375 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:47.449703 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503dc4f9-5ce4-460a-bb3c-6fe264d389e8 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:47.453609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503dc4f9-5ce4-460a-bb3c-6fe264d389e8 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:47.453609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503dc4f9-5ce4-460a-bb3c-6fe264d389e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:47.469847 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-503dc4f9-5ce4-460a-bb3c-6fe264d389e8 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:47.471375 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-503dc4f9-5ce4-460a-bb3c-6fe264d389e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:47.541998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1293/2612] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:47:47.552122 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d182557c-f016-4b39-ad78-69139d7034a9 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:47.552122 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d182557c-f016-4b39-ad78-69139d7034a9 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:47.552317 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d182557c-f016-4b39-ad78-69139d7034a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:47.576822 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-d182557c-f016-4b39-ad78-69139d7034a9 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:47.578267 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-d182557c-f016-4b39-ad78-69139d7034a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:47.587284 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1320/2613] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:47:47.592637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503dc4f9-5ce4-460a-bb3c-6fe264d389e8 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:47.611208 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503dc4f9-5ce4-460a-bb3c-6fe264d389e8 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:47.619132 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88f9ec1-f036-42ff-b266-17cc7c69f5e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:47.620673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88f9ec1-f036-42ff-b266-17cc7c69f5e3 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:47.620869 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88f9ec1-f036-42ff-b266-17cc7c69f5e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:47.621286 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88f9ec1-f036-42ff-b266-17cc7c69f5e3 admin admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:47.622189 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88f9ec1-f036-42ff-b266-17cc7c69f5e3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:47:47.622609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88f9ec1-f036-42ff-b266-17cc7c69f5e3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:47.624322 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88f9ec1-f036-42ff-b266-17cc7c69f5e3 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:47.647443 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1294/2614] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:47:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:47:47.649404 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d182557c-f016-4b39-ad78-69139d7034a9 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:47.659412 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d182557c-f016-4b39-ad78-69139d7034a9 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:47.670574 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a76f27-eb5b-4127-9083-35eada97e613 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:47.672549 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a76f27-eb5b-4127-9083-35eada97e613 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:47.674931 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a76f27-eb5b-4127-9083-35eada97e613 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:47.674931 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8a76f27-eb5b-4127-9083-35eada97e613 admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:47.674931 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a76f27-eb5b-4127-9083-35eada97e613 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:47.677978 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8a76f27-eb5b-4127-9083-35eada97e613 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:47.786633 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1321/2615] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:47:47 2022] POST /identity/v3/projects => generated 367 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:47:47.791026 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88f9ec1-f036-42ff-b266-17cc7c69f5e3 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:47.802808 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88f9ec1-f036-42ff-b266-17cc7c69f5e3 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:47.851726 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:47.868433 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b07c99-884e-4162-b5dd-96623f37bd63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:47.869201 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b07c99-884e-4162-b5dd-96623f37bd63 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:47.869445 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b07c99-884e-4162-b5dd-96623f37bd63 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:47.869734 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b07c99-884e-4162-b5dd-96623f37bd63 admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:47.870455 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b07c99-884e-4162-b5dd-96623f37bd63 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:47.872590 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b07c99-884e-4162-b5dd-96623f37bd63 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:47.983778 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1295/2616] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:47:47 2022] POST /identity/v3/users => generated 284 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:47:47.988138 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a76f27-eb5b-4127-9083-35eada97e613 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:48.000988 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8a76f27-eb5b-4127-9083-35eada97e613 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:48.007180 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-291993cf-cebb-4487-867f-28d965c72d18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:48.007933 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291993cf-cebb-4487-867f-28d965c72d18 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:48.008175 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291993cf-cebb-4487-867f-28d965c72d18 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:48.009342 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291993cf-cebb-4487-867f-28d965c72d18 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:48.010113 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-291993cf-cebb-4487-867f-28d965c72d18 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:48.013189 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-291993cf-cebb-4487-867f-28d965c72d18 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:48.021615 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:DELETE=14,keystone:SELECT=83,keystone:INSERT=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:48.027399 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1322/2617] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:47:47 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:47:48.032434 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b07c99-884e-4162-b5dd-96623f37bd63 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:48.043534 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b07c99-884e-4162-b5dd-96623f37bd63 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:48.048788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70281953-1336-4b2d-b817-c45697461387 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:48.049564 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70281953-1336-4b2d-b817-c45697461387 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:48.049772 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70281953-1336-4b2d-b817-c45697461387 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:48.050054 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70281953-1336-4b2d-b817-c45697461387 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:48.050834 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70281953-1336-4b2d-b817-c45697461387 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:48.052444 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70281953-1336-4b2d-b817-c45697461387 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:48.067273 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1296/2618] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:47:48 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:47:48.069001 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-291993cf-cebb-4487-867f-28d965c72d18 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:48.082112 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-291993cf-cebb-4487-867f-28d965c72d18 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:48.087509 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7f6906-3bb2-4e06-91e6-4938bfa68fff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:48.088474 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7f6906-3bb2-4e06-91e6-4938bfa68fff admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:48.088772 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7f6906-3bb2-4e06-91e6-4938bfa68fff admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:48.089045 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7f6906-3bb2-4e06-91e6-4938bfa68fff admin admin] PATH_INFO: `/v3/projects/2653229ecb3f40f185f2e1b7780ecc49/users/4b96667a707e4d1ea99a26cabd161a2a/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:48.105499 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7f6906-3bb2-4e06-91e6-4938bfa68fff admin admin] RBAC: Authorizing `identity:create_grant(project_id=2653229ecb3f40f185f2e1b7780ecc49, user_id=4b96667a707e4d1ea99a26cabd161a2a, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:48.108807 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7f6906-3bb2-4e06-91e6-4938bfa68fff admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:48.208045 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1323/2619] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:47:48 2022] PUT /identity/v3/projects/2653229ecb3f40f185f2e1b7780ecc49/users/4b96667a707e4d1ea99a26cabd161a2a/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:48.220660 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0905cfe-f381-41a3-84c0-b4dfe9748982 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:48.221119 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0905cfe-f381-41a3-84c0-b4dfe9748982 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:48.221535 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0905cfe-f381-41a3-84c0-b4dfe9748982 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:48.249572 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-e0905cfe-f381-41a3-84c0-b4dfe9748982 None None] MFA Rules not processed for user `4b96667a707e4d1ea99a26cabd161a2a`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:48.252598 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-e0905cfe-f381-41a3-84c0-b4dfe9748982 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:48.315805 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1297/2620] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:48 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:47:48.321303 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7f6906-3bb2-4e06-91e6-4938bfa68fff admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:48.384076 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7f6906-3bb2-4e06-91e6-4938bfa68fff admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:48.389603 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe3ef59-b593-486c-8d59-0db4ba29a40a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:48.390263 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe3ef59-b593-486c-8d59-0db4ba29a40a admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:48.390576 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe3ef59-b593-486c-8d59-0db4ba29a40a admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:48.390861 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe3ef59-b593-486c-8d59-0db4ba29a40a admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:48.391549 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fe3ef59-b593-486c-8d59-0db4ba29a40a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:48.393450 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fe3ef59-b593-486c-8d59-0db4ba29a40a admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:48.482155 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1324/2621] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:47:48 2022] POST /identity/v3/projects => generated 367 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:47:48.486517 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0905cfe-f381-41a3-84c0-b4dfe9748982 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:48.498235 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0905cfe-f381-41a3-84c0-b4dfe9748982 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:48.555559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8757576-5cc8-4afb-a722-6753fb0b6b4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:48.556238 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8757576-5cc8-4afb-a722-6753fb0b6b4c admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:48.556535 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8757576-5cc8-4afb-a722-6753fb0b6b4c admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:48.556821 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8757576-5cc8-4afb-a722-6753fb0b6b4c admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:48.557536 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8757576-5cc8-4afb-a722-6753fb0b6b4c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:48.560182 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8757576-5cc8-4afb-a722-6753fb0b6b4c admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:48.673322 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1298/2622] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:47:48 2022] POST /identity/v3/users => generated 290 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:47:48.675958 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe3ef59-b593-486c-8d59-0db4ba29a40a admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:48.687224 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe3ef59-b593-486c-8d59-0db4ba29a40a admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:48.691980 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a8aeef-a62b-4620-8ff5-9ab98c8d7a43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:48.692503 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a8aeef-a62b-4620-8ff5-9ab98c8d7a43 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:48.696667 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a8aeef-a62b-4620-8ff5-9ab98c8d7a43 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:48.696667 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74a8aeef-a62b-4620-8ff5-9ab98c8d7a43 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:48.696667 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74a8aeef-a62b-4620-8ff5-9ab98c8d7a43 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:48.696667 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74a8aeef-a62b-4620-8ff5-9ab98c8d7a43 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:48.706476 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1325/2623] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:47:48 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 12 17:47:48.710780 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8757576-5cc8-4afb-a722-6753fb0b6b4c admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:48.721107 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8757576-5cc8-4afb-a722-6753fb0b6b4c admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:48.725630 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49a66c2-35c0-41b9-8474-38fb7aa1241c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:48.726556 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49a66c2-35c0-41b9-8474-38fb7aa1241c admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:48.726835 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49a66c2-35c0-41b9-8474-38fb7aa1241c admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:48.727117 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49a66c2-35c0-41b9-8474-38fb7aa1241c admin admin] PATH_INFO: `/v3/projects/0454f017363a4d189ac77cdd9fb348b9/users/f1d6b8101b3c4217acadb47647e8c12d/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:48.742572 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a49a66c2-35c0-41b9-8474-38fb7aa1241c admin admin] RBAC: Authorizing `identity:create_grant(project_id=0454f017363a4d189ac77cdd9fb348b9, user_id=f1d6b8101b3c4217acadb47647e8c12d, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:48.745279 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a49a66c2-35c0-41b9-8474-38fb7aa1241c admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:48.864019 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1299/2624] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:47:48 2022] PUT /identity/v3/projects/0454f017363a4d189ac77cdd9fb348b9/users/f1d6b8101b3c4217acadb47647e8c12d/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:47:48.867966 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a8aeef-a62b-4620-8ff5-9ab98c8d7a43 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:48.879057 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74a8aeef-a62b-4620-8ff5-9ab98c8d7a43 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:48.935734 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd0b60c-ce27-46c5-ab19-d9e0fe3a6264 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:48.936394 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd0b60c-ce27-46c5-ab19-d9e0fe3a6264 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:48.936681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd0b60c-ce27-46c5-ab19-d9e0fe3a6264 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:48.936959 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd0b60c-ce27-46c5-ab19-d9e0fe3a6264 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:48.937685 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd0b60c-ce27-46c5-ab19-d9e0fe3a6264 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:48.939308 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd0b60c-ce27-46c5-ab19-d9e0fe3a6264 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:48.949514 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1326/2625] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:47:48 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:47:48.953657 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49a66c2-35c0-41b9-8474-38fb7aa1241c admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:48.963915 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49a66c2-35c0-41b9-8474-38fb7aa1241c admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:48.969681 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e80725f-46b7-4300-890a-f0d58140bcf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:48.970837 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e80725f-46b7-4300-890a-f0d58140bcf9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:48.971140 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e80725f-46b7-4300-890a-f0d58140bcf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:48.971559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e80725f-46b7-4300-890a-f0d58140bcf9 admin admin] PATH_INFO: `/v3/projects/0454f017363a4d189ac77cdd9fb348b9/users/f1d6b8101b3c4217acadb47647e8c12d/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:48.974073 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e80725f-46b7-4300-890a-f0d58140bcf9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0454f017363a4d189ac77cdd9fb348b9, user_id=f1d6b8101b3c4217acadb47647e8c12d, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:48.977174 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e80725f-46b7-4300-890a-f0d58140bcf9 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:48.988797 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1300/2626] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:47:48 2022] PUT /identity/v3/projects/0454f017363a4d189ac77cdd9fb348b9/users/f1d6b8101b3c4217acadb47647e8c12d/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:49.001416 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216d7fde-2620-460b-b9f2-6e5c61d39fea None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:49.001714 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216d7fde-2620-460b-b9f2-6e5c61d39fea None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:49.001998 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216d7fde-2620-460b-b9f2-6e5c61d39fea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:49.030332 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-216d7fde-2620-460b-b9f2-6e5c61d39fea None None] MFA Rules not processed for user `f1d6b8101b3c4217acadb47647e8c12d`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:49.031595 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-216d7fde-2620-460b-b9f2-6e5c61d39fea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:49.066245 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:49.100550 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1327/2627] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:48 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:47:49.113348 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e80725f-46b7-4300-890a-f0d58140bcf9 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:49.118619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-2e80725f-46b7-4300-890a-f0d58140bcf9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:49.120572 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:49.176718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e80725f-46b7-4300-890a-f0d58140bcf9 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:49.180329 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1896462e-7ce7-409c-ba6b-7f7b55a9430f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:49.181008 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1896462e-7ce7-409c-ba6b-7f7b55a9430f service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:49.181302 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1896462e-7ce7-409c-ba6b-7f7b55a9430f service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:49.182274 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1896462e-7ce7-409c-ba6b-7f7b55a9430f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:49.192125 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1896462e-7ce7-409c-ba6b-7f7b55a9430f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:49.193828 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1896462e-7ce7-409c-ba6b-7f7b55a9430f service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:49.204439 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1301/2628] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:47:49 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:47:49.420743 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216d7fde-2620-460b-b9f2-6e5c61d39fea None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:49.430399 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216d7fde-2620-460b-b9f2-6e5c61d39fea None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:49.434028 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a4d06a-e4aa-4f11-ac17-43ebe11a25fb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:49.434805 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a4d06a-e4aa-4f11-ac17-43ebe11a25fb service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:49.435101 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a4d06a-e4aa-4f11-ac17-43ebe11a25fb service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:49.435389 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a4d06a-e4aa-4f11-ac17-43ebe11a25fb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:49.492289 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a4d06a-e4aa-4f11-ac17-43ebe11a25fb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:49.493739 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a4d06a-e4aa-4f11-ac17-43ebe11a25fb service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:49.503066 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1328/2629] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:47:49 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:47:56.101295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1896462e-7ce7-409c-ba6b-7f7b55a9430f service cinder] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:56.102853 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-1896462e-7ce7-409c-ba6b-7f7b55a9430f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:56.178320 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1896462e-7ce7-409c-ba6b-7f7b55a9430f service cinder] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:56.182981 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95fadc3b-7edd-4549-a9bf-106f573d7936 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:56.183889 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fadc3b-7edd-4549-a9bf-106f573d7936 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:56.184182 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fadc3b-7edd-4549-a9bf-106f573d7936 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:56.184465 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fadc3b-7edd-4549-a9bf-106f573d7936 admin admin] PATH_INFO: `/v3/users/e4413ac8b045432cb9ea8a156fa12571` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:56.186333 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95fadc3b-7edd-4549-a9bf-106f573d7936 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e4413ac8b045432cb9ea8a156fa12571)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:56.188357 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95fadc3b-7edd-4549-a9bf-106f573d7936 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:56.393107 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-95fadc3b-7edd-4549-a9bf-106f573d7936 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e4413ac8b045432cb9ea8a156fa12571'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:56.482043 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-95fadc3b-7edd-4549-a9bf-106f573d7936 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e4413ac8b045432cb9ea8a156fa12571'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:56.483285 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-95fadc3b-7edd-4549-a9bf-106f573d7936 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e4413ac8b045432cb9ea8a156fa12571'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:56.492589 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-95fadc3b-7edd-4549-a9bf-106f573d7936 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e4413ac8b045432cb9ea8a156fa12571'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:56.505297 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-95fadc3b-7edd-4549-a9bf-106f573d7936 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e4413ac8b045432cb9ea8a156fa12571'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:56.511235 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1302/2630] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:47:56 2022] DELETE /identity/v3/users/e4413ac8b045432cb9ea8a156fa12571 => generated 0 bytes in 410 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:56.515881 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df1ecbb-9929-4199-9ddb-e55c4296ce65 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:56.516368 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df1ecbb-9929-4199-9ddb-e55c4296ce65 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:56.516467 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df1ecbb-9929-4199-9ddb-e55c4296ce65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:56.534079 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-3df1ecbb-9929-4199-9ddb-e55c4296ce65 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:56.535320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-3df1ecbb-9929-4199-9ddb-e55c4296ce65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:56.596461 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1329/2631] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:47:56.606004 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95fadc3b-7edd-4549-a9bf-106f573d7936 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:56.608053 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-95fadc3b-7edd-4549-a9bf-106f573d7936 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:56.636727 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:56.667094 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95fadc3b-7edd-4549-a9bf-106f573d7936 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:56.672163 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8edde0-2c33-48ee-ac5d-8d80e291ea1e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:56.672861 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8edde0-2c33-48ee-ac5d-8d80e291ea1e service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:56.673154 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8edde0-2c33-48ee-ac5d-8d80e291ea1e service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:56.673461 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8edde0-2c33-48ee-ac5d-8d80e291ea1e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:56.682942 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8edde0-2c33-48ee-ac5d-8d80e291ea1e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:56.684422 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8edde0-2c33-48ee-ac5d-8d80e291ea1e service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:56.694649 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1303/2632] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:47:56 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:47:57.056204 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df1ecbb-9929-4199-9ddb-e55c4296ce65 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:57.057607 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-3df1ecbb-9929-4199-9ddb-e55c4296ce65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:57.069845 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df1ecbb-9929-4199-9ddb-e55c4296ce65 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:57.074344 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5a18b8-0185-4ca2-abab-0e691d858c70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:57.075242 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5a18b8-0185-4ca2-abab-0e691d858c70 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:57.075526 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5a18b8-0185-4ca2-abab-0e691d858c70 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:57.075801 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5a18b8-0185-4ca2-abab-0e691d858c70 admin admin] PATH_INFO: `/v3/users/4b96667a707e4d1ea99a26cabd161a2a` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:57.077623 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c5a18b8-0185-4ca2-abab-0e691d858c70 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4b96667a707e4d1ea99a26cabd161a2a)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:57.079570 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c5a18b8-0185-4ca2-abab-0e691d858c70 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:57.116471 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8edde0-2c33-48ee-ac5d-8d80e291ea1e service neutron] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:57.118550 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-5c8edde0-2c33-48ee-ac5d-8d80e291ea1e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:57.131599 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8edde0-2c33-48ee-ac5d-8d80e291ea1e service neutron] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:57.136383 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd17616f-5a65-4f25-a7c4-06c895e8d86c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:57.137309 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd17616f-5a65-4f25-a7c4-06c895e8d86c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:57.137578 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd17616f-5a65-4f25-a7c4-06c895e8d86c admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:57.137860 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd17616f-5a65-4f25-a7c4-06c895e8d86c admin admin] PATH_INFO: `/v3/projects/682341e3a9254b3e82adc1bf50a7462d` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:57.139852 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd17616f-5a65-4f25-a7c4-06c895e8d86c admin admin] RBAC: Authorizing `identity:delete_project(project_id=682341e3a9254b3e82adc1bf50a7462d)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:57.141952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd17616f-5a65-4f25-a7c4-06c895e8d86c admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:57.261623 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-7c5a18b8-0185-4ca2-abab-0e691d858c70 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4b96667a707e4d1ea99a26cabd161a2a'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:57.262283 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-7c5a18b8-0185-4ca2-abab-0e691d858c70 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4b96667a707e4d1ea99a26cabd161a2a'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:57.263007 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-7c5a18b8-0185-4ca2-abab-0e691d858c70 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4b96667a707e4d1ea99a26cabd161a2a'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:57.271334 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-7c5a18b8-0185-4ca2-abab-0e691d858c70 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4b96667a707e4d1ea99a26cabd161a2a'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:57.278929 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=92,keystone:DELETE=8 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:57.285352 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-7c5a18b8-0185-4ca2-abab-0e691d858c70 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4b96667a707e4d1ea99a26cabd161a2a'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:57.289136 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-fd17616f-5a65-4f25-a7c4-06c895e8d86c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '682341e3a9254b3e82adc1bf50a7462d', 'request_id': 'req-fd17616f-5a65-4f25-a7c4-06c895e8d86c'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:57.295698 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-fd17616f-5a65-4f25-a7c4-06c895e8d86c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '682341e3a9254b3e82adc1bf50a7462d', 'request_id': 'req-fd17616f-5a65-4f25-a7c4-06c895e8d86c'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:57.301243 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-fd17616f-5a65-4f25-a7c4-06c895e8d86c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '682341e3a9254b3e82adc1bf50a7462d', 'request_id': 'req-fd17616f-5a65-4f25-a7c4-06c895e8d86c'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:57.355752 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1330/2633] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:47:57 2022] DELETE /identity/v3/users/4b96667a707e4d1ea99a26cabd161a2a => generated 0 bytes in 300 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:57.360573 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f80b83-79ce-41fb-8ebd-4619e8486219 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:57.360867 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f80b83-79ce-41fb-8ebd-4619e8486219 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:57.361153 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f80b83-79ce-41fb-8ebd-4619e8486219 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:57.379325 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-61f80b83-79ce-41fb-8ebd-4619e8486219 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:57.380543 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-61f80b83-79ce-41fb-8ebd-4619e8486219 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:57.414453 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-fd17616f-5a65-4f25-a7c4-06c895e8d86c admin admin] The token cache is being invalidate because project 682341e3a9254b3e82adc1bf50a7462d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:47:57.414926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-fd17616f-5a65-4f25-a7c4-06c895e8d86c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:57.415849 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-fd17616f-5a65-4f25-a7c4-06c895e8d86c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:57.417485 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1304/2634] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:47:57 2022] DELETE /identity/v3/projects/682341e3a9254b3e82adc1bf50a7462d => generated 0 bytes in 302 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:47:57.421493 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd17616f-5a65-4f25-a7c4-06c895e8d86c admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:57.422896 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-fd17616f-5a65-4f25-a7c4-06c895e8d86c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:57.442960 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1331/2635] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:47:57.454183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f80b83-79ce-41fb-8ebd-4619e8486219 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:57.455691 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-61f80b83-79ce-41fb-8ebd-4619e8486219 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:57.489462 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd17616f-5a65-4f25-a7c4-06c895e8d86c admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:57.494083 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572a735a-7d25-4257-95bb-8d60c33a42ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:57.495002 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572a735a-7d25-4257-95bb-8d60c33a42ca admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:57.495286 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572a735a-7d25-4257-95bb-8d60c33a42ca admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:57.495573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572a735a-7d25-4257-95bb-8d60c33a42ca admin admin] PATH_INFO: `/v3/users/e357115b18624b00a58524079d297b0c` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:57.496779 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-572a735a-7d25-4257-95bb-8d60c33a42ca admin admin] RBAC: Authorizing `identity:delete_user(user_id=e357115b18624b00a58524079d297b0c)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:57.499245 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-572a735a-7d25-4257-95bb-8d60c33a42ca admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:57.516451 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f80b83-79ce-41fb-8ebd-4619e8486219 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:57.521151 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0780bc16-377c-4ff7-8645-0ec296870cce service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:57.521912 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0780bc16-377c-4ff7-8645-0ec296870cce service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:57.522206 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0780bc16-377c-4ff7-8645-0ec296870cce service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:57.522541 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0780bc16-377c-4ff7-8645-0ec296870cce service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:57.534051 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0780bc16-377c-4ff7-8645-0ec296870cce service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:57.535652 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0780bc16-377c-4ff7-8645-0ec296870cce service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:57.545803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1332/2636] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:47:57 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:47:57.702638 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-572a735a-7d25-4257-95bb-8d60c33a42ca admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e357115b18624b00a58524079d297b0c'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:57.706580 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:57.768030 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-572a735a-7d25-4257-95bb-8d60c33a42ca admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e357115b18624b00a58524079d297b0c'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:57.768892 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-572a735a-7d25-4257-95bb-8d60c33a42ca admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e357115b18624b00a58524079d297b0c'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:57.776918 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-572a735a-7d25-4257-95bb-8d60c33a42ca admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e357115b18624b00a58524079d297b0c'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:57.790171 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-572a735a-7d25-4257-95bb-8d60c33a42ca admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e357115b18624b00a58524079d297b0c'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:57.797252 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1305/2637] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:47:57 2022] DELETE /identity/v3/users/e357115b18624b00a58524079d297b0c => generated 0 bytes in 376 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:57.952242 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0780bc16-377c-4ff7-8645-0ec296870cce service neutron] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:57.953741 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-0780bc16-377c-4ff7-8645-0ec296870cce service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:57.969362 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0780bc16-377c-4ff7-8645-0ec296870cce service neutron] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:58.031688 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8378c9d-f780-46cc-8a80-c5f4df2b6028 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:58.032591 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8378c9d-f780-46cc-8a80-c5f4df2b6028 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:58.032911 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8378c9d-f780-46cc-8a80-c5f4df2b6028 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:58.033224 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8378c9d-f780-46cc-8a80-c5f4df2b6028 admin admin] PATH_INFO: `/v3/projects/2653229ecb3f40f185f2e1b7780ecc49` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:58.034742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8378c9d-f780-46cc-8a80-c5f4df2b6028 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2653229ecb3f40f185f2e1b7780ecc49)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:58.036448 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8378c9d-f780-46cc-8a80-c5f4df2b6028 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:58.148255 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:58.165443 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-f8378c9d-f780-46cc-8a80-c5f4df2b6028 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2653229ecb3f40f185f2e1b7780ecc49', 'request_id': 'req-f8378c9d-f780-46cc-8a80-c5f4df2b6028'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:58.171798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-f8378c9d-f780-46cc-8a80-c5f4df2b6028 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2653229ecb3f40f185f2e1b7780ecc49', 'request_id': 'req-f8378c9d-f780-46cc-8a80-c5f4df2b6028'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:58.236117 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-f8378c9d-f780-46cc-8a80-c5f4df2b6028 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2653229ecb3f40f185f2e1b7780ecc49', 'request_id': 'req-f8378c9d-f780-46cc-8a80-c5f4df2b6028'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:58.249267 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-f8378c9d-f780-46cc-8a80-c5f4df2b6028 admin admin] The token cache is being invalidate because project 2653229ecb3f40f185f2e1b7780ecc49 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:47:58.249736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-f8378c9d-f780-46cc-8a80-c5f4df2b6028 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:58.250562 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-f8378c9d-f780-46cc-8a80-c5f4df2b6028 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:58.252118 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1333/2638] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:47:57 2022] DELETE /identity/v3/projects/2653229ecb3f40f185f2e1b7780ecc49 => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:58.256192 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572a735a-7d25-4257-95bb-8d60c33a42ca admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:58.257727 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-572a735a-7d25-4257-95bb-8d60c33a42ca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:58.272654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572a735a-7d25-4257-95bb-8d60c33a42ca admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:58.333142 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e28bd8-4ac8-40c7-9d7b-4fd89ca34cf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:58.334028 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e28bd8-4ac8-40c7-9d7b-4fd89ca34cf8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:58.334314 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e28bd8-4ac8-40c7-9d7b-4fd89ca34cf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:58.334623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e28bd8-4ac8-40c7-9d7b-4fd89ca34cf8 admin admin] PATH_INFO: `/v3/users/f1d6b8101b3c4217acadb47647e8c12d` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:58.335846 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e28bd8-4ac8-40c7-9d7b-4fd89ca34cf8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f1d6b8101b3c4217acadb47647e8c12d)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:58.337757 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e28bd8-4ac8-40c7-9d7b-4fd89ca34cf8 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:58.366958 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8378c9d-f780-46cc-8a80-c5f4df2b6028 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:58.366958 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-f8378c9d-f780-46cc-8a80-c5f4df2b6028 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:58.379015 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8378c9d-f780-46cc-8a80-c5f4df2b6028 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:58.384483 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593451e9-ccd3-4236-832b-95ff37d29863 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:58.385460 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593451e9-ccd3-4236-832b-95ff37d29863 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:58.385742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593451e9-ccd3-4236-832b-95ff37d29863 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:58.386014 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593451e9-ccd3-4236-832b-95ff37d29863 admin admin] PATH_INFO: `/v3/projects/2e2557e412f24b92a6cfc6edcfa306d7` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:58.387336 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-593451e9-ccd3-4236-832b-95ff37d29863 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2e2557e412f24b92a6cfc6edcfa306d7)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:58.389329 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-593451e9-ccd3-4236-832b-95ff37d29863 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:58.557618 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-e0e28bd8-4ac8-40c7-9d7b-4fd89ca34cf8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f1d6b8101b3c4217acadb47647e8c12d'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:58.557978 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-593451e9-ccd3-4236-832b-95ff37d29863 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2e2557e412f24b92a6cfc6edcfa306d7', 'request_id': 'req-593451e9-ccd3-4236-832b-95ff37d29863'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:58.565667 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-593451e9-ccd3-4236-832b-95ff37d29863 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2e2557e412f24b92a6cfc6edcfa306d7', 'request_id': 'req-593451e9-ccd3-4236-832b-95ff37d29863'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:58.630249 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-e0e28bd8-4ac8-40c7-9d7b-4fd89ca34cf8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f1d6b8101b3c4217acadb47647e8c12d'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:58.630954 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-e0e28bd8-4ac8-40c7-9d7b-4fd89ca34cf8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f1d6b8101b3c4217acadb47647e8c12d'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:58.639726 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-e0e28bd8-4ac8-40c7-9d7b-4fd89ca34cf8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f1d6b8101b3c4217acadb47647e8c12d'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:58.653798 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-e0e28bd8-4ac8-40c7-9d7b-4fd89ca34cf8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f1d6b8101b3c4217acadb47647e8c12d'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:58.661954 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1306/2639] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:47:58 2022] DELETE /identity/v3/users/f1d6b8101b3c4217acadb47647e8c12d => generated 0 bytes in 406 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:47:58.722345 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-593451e9-ccd3-4236-832b-95ff37d29863 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2e2557e412f24b92a6cfc6edcfa306d7', 'request_id': 'req-593451e9-ccd3-4236-832b-95ff37d29863'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:58.735635 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-593451e9-ccd3-4236-832b-95ff37d29863 admin admin] The token cache is being invalidate because project 2e2557e412f24b92a6cfc6edcfa306d7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:47:58.735929 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-593451e9-ccd3-4236-832b-95ff37d29863 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:58.736606 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-593451e9-ccd3-4236-832b-95ff37d29863 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:58.738202 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1334/2640] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:47:58 2022] DELETE /identity/v3/projects/2e2557e412f24b92a6cfc6edcfa306d7 => generated 0 bytes in 376 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:47:58.743364 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fff7839-d0dd-4e47-8ac4-8b275e31529c None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:58.743669 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fff7839-d0dd-4e47-8ac4-8b275e31529c None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:58.744004 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fff7839-d0dd-4e47-8ac4-8b275e31529c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:58.762743 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-6fff7839-d0dd-4e47-8ac4-8b275e31529c None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:58.764272 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-6fff7839-d0dd-4e47-8ac4-8b275e31529c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:58.812308 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:58.828320 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1307/2641] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:58 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:47:58.837696 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33b2135-95fe-497b-8cc6-8e7346fcc992 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:58.838057 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33b2135-95fe-497b-8cc6-8e7346fcc992 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:58.838450 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e33b2135-95fe-497b-8cc6-8e7346fcc992 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:58.858949 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-e33b2135-95fe-497b-8cc6-8e7346fcc992 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:58.860198 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-e33b2135-95fe-497b-8cc6-8e7346fcc992 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:58.869244 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1335/2642] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:58 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:47:58.873319 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fff7839-d0dd-4e47-8ac4-8b275e31529c None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:58.883411 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fff7839-d0dd-4e47-8ac4-8b275e31529c None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:58.888754 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30899604-9a8f-48bc-bf2b-ecd6771d60de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:58.889528 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30899604-9a8f-48bc-bf2b-ecd6771d60de admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:58.889881 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30899604-9a8f-48bc-bf2b-ecd6771d60de admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:58.890228 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30899604-9a8f-48bc-bf2b-ecd6771d60de admin admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:58.891081 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30899604-9a8f-48bc-bf2b-ecd6771d60de admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:47:58.891392 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30899604-9a8f-48bc-bf2b-ecd6771d60de admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:58.892983 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30899604-9a8f-48bc-bf2b-ecd6771d60de admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:58.909565 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1308/2643] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:47:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:47:58.910811 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33b2135-95fe-497b-8cc6-8e7346fcc992 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:58.921343 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e33b2135-95fe-497b-8cc6-8e7346fcc992 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:58.927058 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2352f452-2e3f-458e-a44a-77aaee7d548a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:58.927872 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2352f452-2e3f-458e-a44a-77aaee7d548a admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:58.928288 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2352f452-2e3f-458e-a44a-77aaee7d548a admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:58.928633 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2352f452-2e3f-458e-a44a-77aaee7d548a admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:58.929397 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2352f452-2e3f-458e-a44a-77aaee7d548a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:58.931359 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2352f452-2e3f-458e-a44a-77aaee7d548a admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:59.049989 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1336/2644] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:47:58 2022] POST /identity/v3/projects => generated 377 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:47:59.054378 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30899604-9a8f-48bc-bf2b-ecd6771d60de admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:59.064725 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30899604-9a8f-48bc-bf2b-ecd6771d60de admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:59.089987 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2352f452-2e3f-458e-a44a-77aaee7d548a admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:59.092633 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-2352f452-2e3f-458e-a44a-77aaee7d548a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:59.130269 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc8700a-671e-4466-952b-4498950f3d17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:59.130983 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc8700a-671e-4466-952b-4498950f3d17 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:59.131292 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc8700a-671e-4466-952b-4498950f3d17 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:59.131570 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc8700a-671e-4466-952b-4498950f3d17 admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:59.132279 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc8700a-671e-4466-952b-4498950f3d17 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:59.134180 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc8700a-671e-4466-952b-4498950f3d17 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:59.163817 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2352f452-2e3f-458e-a44a-77aaee7d548a admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:59.168670 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96df4408-29ca-488e-a41d-92ff358ad972 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:59.169528 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96df4408-29ca-488e-a41d-92ff358ad972 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:59.169820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96df4408-29ca-488e-a41d-92ff358ad972 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:59.170177 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96df4408-29ca-488e-a41d-92ff358ad972 admin admin] PATH_INFO: `/v3/projects/0454f017363a4d189ac77cdd9fb348b9` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:59.171489 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96df4408-29ca-488e-a41d-92ff358ad972 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0454f017363a4d189ac77cdd9fb348b9)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:59.173586 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96df4408-29ca-488e-a41d-92ff358ad972 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:59.186012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:59.219951 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1309/2645] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:47:59 2022] POST /identity/v3/users => generated 289 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:47:59.223900 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc8700a-671e-4466-952b-4498950f3d17 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:59.235217 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc8700a-671e-4466-952b-4498950f3d17 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:59.239981 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783356e0-f499-4549-8a9d-a7d0ded22340 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:59.240725 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783356e0-f499-4549-8a9d-a7d0ded22340 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:59.241017 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783356e0-f499-4549-8a9d-a7d0ded22340 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:59.241302 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783356e0-f499-4549-8a9d-a7d0ded22340 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:59.242034 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-783356e0-f499-4549-8a9d-a7d0ded22340 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:59.243673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-783356e0-f499-4549-8a9d-a7d0ded22340 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:59.254127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1310/2646] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:47:59 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 12 17:47:59.257982 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783356e0-f499-4549-8a9d-a7d0ded22340 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:59.268587 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783356e0-f499-4549-8a9d-a7d0ded22340 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:59.273675 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b5b6f7-3bdd-4e1d-a7d9-80b9110e6fe0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:59.274322 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b5b6f7-3bdd-4e1d-a7d9-80b9110e6fe0 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:59.277785 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b5b6f7-3bdd-4e1d-a7d9-80b9110e6fe0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:59.277785 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b5b6f7-3bdd-4e1d-a7d9-80b9110e6fe0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:59.277785 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b5b6f7-3bdd-4e1d-a7d9-80b9110e6fe0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:59.277785 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b5b6f7-3bdd-4e1d-a7d9-80b9110e6fe0 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:59.288839 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1311/2647] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:47:59 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:47:59.292684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b5b6f7-3bdd-4e1d-a7d9-80b9110e6fe0 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:59.302383 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b5b6f7-3bdd-4e1d-a7d9-80b9110e6fe0 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:59.315397 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-96df4408-29ca-488e-a41d-92ff358ad972 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0454f017363a4d189ac77cdd9fb348b9', 'request_id': 'req-96df4408-29ca-488e-a41d-92ff358ad972'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:59.321022 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-96df4408-29ca-488e-a41d-92ff358ad972 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0454f017363a4d189ac77cdd9fb348b9', 'request_id': 'req-96df4408-29ca-488e-a41d-92ff358ad972'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:59.361993 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b177cf55-1dc8-48aa-ae1b-cf21e32f75a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:59.362916 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b177cf55-1dc8-48aa-ae1b-cf21e32f75a1 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:59.363207 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b177cf55-1dc8-48aa-ae1b-cf21e32f75a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:59.363498 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b177cf55-1dc8-48aa-ae1b-cf21e32f75a1 admin admin] PATH_INFO: `/v3/projects/3594d8c4a6bc4fdfb378c509820837ac/users/8810a50afb284cdb9dff1b23d3244375/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:59.378636 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b177cf55-1dc8-48aa-ae1b-cf21e32f75a1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3594d8c4a6bc4fdfb378c509820837ac, user_id=8810a50afb284cdb9dff1b23d3244375, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:59.381295 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b177cf55-1dc8-48aa-ae1b-cf21e32f75a1 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:59.382624 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-96df4408-29ca-488e-a41d-92ff358ad972 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0454f017363a4d189ac77cdd9fb348b9', 'request_id': 'req-96df4408-29ca-488e-a41d-92ff358ad972'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:59.395069 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-96df4408-29ca-488e-a41d-92ff358ad972 admin admin] The token cache is being invalidate because project 0454f017363a4d189ac77cdd9fb348b9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:47:59.395366 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-96df4408-29ca-488e-a41d-92ff358ad972 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:59.396059 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-96df4408-29ca-488e-a41d-92ff358ad972 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:47:59.397626 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1337/2648] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:47:59 2022] DELETE /identity/v3/projects/0454f017363a4d189ac77cdd9fb348b9 => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:47:59.402256 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96e9480-d89a-4ec9-8a46-f8ff855a247e None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:59.402587 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96e9480-d89a-4ec9-8a46-f8ff855a247e None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:59.402872 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96e9480-d89a-4ec9-8a46-f8ff855a247e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:59.422381 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-f96e9480-d89a-4ec9-8a46-f8ff855a247e None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:59.423612 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-f96e9480-d89a-4ec9-8a46-f8ff855a247e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:59.431563 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1338/2649] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:59 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:47:59.439569 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f341c48a-dd68-4811-a67e-7cd6dabca4a1 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:59.439984 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f341c48a-dd68-4811-a67e-7cd6dabca4a1 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:59.444262 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f341c48a-dd68-4811-a67e-7cd6dabca4a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:59.447879 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1312/2650] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:47:59 2022] PUT /identity/v3/projects/3594d8c4a6bc4fdfb378c509820837ac/users/8810a50afb284cdb9dff1b23d3244375/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:47:59.458018 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055028f0-a2b7-461c-815e-f150e71ebcf6 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:59.458330 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055028f0-a2b7-461c-815e-f150e71ebcf6 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:59.458651 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055028f0-a2b7-461c-815e-f150e71ebcf6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:59.459779 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-f341c48a-dd68-4811-a67e-7cd6dabca4a1 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:59.461130 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-f341c48a-dd68-4811-a67e-7cd6dabca4a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:59.487618 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-055028f0-a2b7-461c-815e-f150e71ebcf6 None None] MFA Rules not processed for user `8810a50afb284cdb9dff1b23d3244375`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:47:59.488891 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-055028f0-a2b7-461c-815e-f150e71ebcf6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:59.505939 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:47:59.524220 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1339/2651] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:59 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:47:59.528488 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f341c48a-dd68-4811-a67e-7cd6dabca4a1 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:59.538415 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f341c48a-dd68-4811-a67e-7cd6dabca4a1 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:59.543318 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68c2f42-798a-4a64-81ff-c16e664f0105 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:59.543978 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68c2f42-798a-4a64-81ff-c16e664f0105 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:59.544272 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68c2f42-798a-4a64-81ff-c16e664f0105 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:59.544554 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68c2f42-798a-4a64-81ff-c16e664f0105 admin admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:59.546000 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68c2f42-798a-4a64-81ff-c16e664f0105 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:47:59.546349 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68c2f42-798a-4a64-81ff-c16e664f0105 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:59.547939 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68c2f42-798a-4a64-81ff-c16e664f0105 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:59.559906 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1340/2652] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:47:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:47:59.569027 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68c2f42-798a-4a64-81ff-c16e664f0105 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:59.569392 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1313/2653] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:47:59 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:47:59.580727 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68c2f42-798a-4a64-81ff-c16e664f0105 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:59.583467 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055028f0-a2b7-461c-815e-f150e71ebcf6 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:59.585337 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-055028f0-a2b7-461c-815e-f150e71ebcf6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:47:59.585586 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5ca37f-c1fa-4663-aaea-530293e2aab5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:59.586246 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5ca37f-c1fa-4663-aaea-530293e2aab5 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:59.586577 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5ca37f-c1fa-4663-aaea-530293e2aab5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:59.586859 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5ca37f-c1fa-4663-aaea-530293e2aab5 admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:59.587555 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5ca37f-c1fa-4663-aaea-530293e2aab5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:59.589552 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5ca37f-c1fa-4663-aaea-530293e2aab5 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:59.643813 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055028f0-a2b7-461c-815e-f150e71ebcf6 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:59.647575 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f2f080-4683-4684-8006-b132fc9d5315 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4018d94a1feb4d6dade88ee9663f9729', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:59.648230 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f2f080-4683-4684-8006-b132fc9d5315 service glance] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:59.648522 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f2f080-4683-4684-8006-b132fc9d5315 service glance] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:59.648808 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f2f080-4683-4684-8006-b132fc9d5315 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:59.657475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4f2f080-4683-4684-8006-b132fc9d5315 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:59.658949 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4f2f080-4683-4684-8006-b132fc9d5315 service glance] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:59.668125 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1314/2654] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:47:59 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:47:59.698268 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1341/2655] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:47:59 2022] POST /identity/v3/projects => generated 371 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:47:59.699610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f2f080-4683-4684-8006-b132fc9d5315 service glance] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:59.712037 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f2f080-4683-4684-8006-b132fc9d5315 service glance] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:59.774254 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ac80ca-d2ac-4da8-b294-d256691a8916 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:59.775085 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ac80ca-d2ac-4da8-b294-d256691a8916 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:59.775378 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ac80ca-d2ac-4da8-b294-d256691a8916 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:59.775656 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ac80ca-d2ac-4da8-b294-d256691a8916 admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:59.776348 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17ac80ca-d2ac-4da8-b294-d256691a8916 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:59.778243 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17ac80ca-d2ac-4da8-b294-d256691a8916 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:59.848385 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1315/2656] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:47:59 2022] POST /identity/v3/users => generated 286 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:47:59.852556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5ca37f-c1fa-4663-aaea-530293e2aab5 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:59.864051 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5ca37f-c1fa-4663-aaea-530293e2aab5 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:59.869178 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e744438a-b2c3-44a6-8061-699935b820e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:59.869953 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e744438a-b2c3-44a6-8061-699935b820e7 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:59.870250 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e744438a-b2c3-44a6-8061-699935b820e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:59.870573 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e744438a-b2c3-44a6-8061-699935b820e7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:59.871316 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e744438a-b2c3-44a6-8061-699935b820e7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:59.872909 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e744438a-b2c3-44a6-8061-699935b820e7 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:59.884119 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1342/2657] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:47:59 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:47:59.888484 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ac80ca-d2ac-4da8-b294-d256691a8916 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:59.898725 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ac80ca-d2ac-4da8-b294-d256691a8916 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:59.903376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d49fe5-ead9-4b7b-a689-938ab4b30ad6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:59.904046 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d49fe5-ead9-4b7b-a689-938ab4b30ad6 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:59.904387 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d49fe5-ead9-4b7b-a689-938ab4b30ad6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:59.904678 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d49fe5-ead9-4b7b-a689-938ab4b30ad6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:59.905419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d49fe5-ead9-4b7b-a689-938ab4b30ad6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:59.907050 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d49fe5-ead9-4b7b-a689-938ab4b30ad6 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:47:59.917300 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1316/2658] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:47:59 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:47:59.922120 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e744438a-b2c3-44a6-8061-699935b820e7 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:47:59.932804 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e744438a-b2c3-44a6-8061-699935b820e7 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:47:59.938183 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0728a082-51d6-44cd-8b71-3f203a4ca936 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:47:59.939098 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0728a082-51d6-44cd-8b71-3f203a4ca936 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:47:59.939386 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0728a082-51d6-44cd-8b71-3f203a4ca936 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:47:59.939670 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0728a082-51d6-44cd-8b71-3f203a4ca936 admin admin] PATH_INFO: `/v3/projects/dcfce527d1fa46f398920d05eab9ae78/users/53deec137af84c3fbba5d6807a5ca208/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:47:59.955731 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0728a082-51d6-44cd-8b71-3f203a4ca936 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dcfce527d1fa46f398920d05eab9ae78, user_id=53deec137af84c3fbba5d6807a5ca208, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:47:59.958613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0728a082-51d6-44cd-8b71-3f203a4ca936 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:00.041831 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1343/2659] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:47:59 2022] PUT /identity/v3/projects/dcfce527d1fa46f398920d05eab9ae78/users/53deec137af84c3fbba5d6807a5ca208/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:48:00.047880 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f2e852-40c5-4d0f-a722-9fabc5d6b08b None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:00.048208 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f2e852-40c5-4d0f-a722-9fabc5d6b08b None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:00.048499 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f2e852-40c5-4d0f-a722-9fabc5d6b08b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:00.067045 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-c1f2e852-40c5-4d0f-a722-9fabc5d6b08b None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:48:00.068587 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-c1f2e852-40c5-4d0f-a722-9fabc5d6b08b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:00.101693 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:48:00.129321 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1317/2660] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:48:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:48:00.140032 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0728a082-51d6-44cd-8b71-3f203a4ca936 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:00.141899 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-0728a082-51d6-44cd-8b71-3f203a4ca936 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:00.172091 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:48:00.204602 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0728a082-51d6-44cd-8b71-3f203a4ca936 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:00.208623 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755875ea-cb09-4f9f-8608-65f18e992508 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:00.209482 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755875ea-cb09-4f9f-8608-65f18e992508 service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:00.209997 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755875ea-cb09-4f9f-8608-65f18e992508 service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:00.210454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755875ea-cb09-4f9f-8608-65f18e992508 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:00.221624 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-755875ea-cb09-4f9f-8608-65f18e992508 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:00.223122 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-755875ea-cb09-4f9f-8608-65f18e992508 service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:00.234137 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1344/2661] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:48:00 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:48:01.449214 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806817a6-1811-41ff-bab8-0b882e5ba8b9 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:01.450212 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806817a6-1811-41ff-bab8-0b882e5ba8b9 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:01.450534 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806817a6-1811-41ff-bab8-0b882e5ba8b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:01.482141 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-806817a6-1811-41ff-bab8-0b882e5ba8b9 None None] MFA Rules not processed for user `53deec137af84c3fbba5d6807a5ca208`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:48:01.484506 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-806817a6-1811-41ff-bab8-0b882e5ba8b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:01.557692 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1318/2662] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:48:01 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:48:01.563248 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755875ea-cb09-4f9f-8608-65f18e992508 service neutron] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:01.576253 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755875ea-cb09-4f9f-8608-65f18e992508 service neutron] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:01.583685 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8afdee8-e133-4605-9860-3c06674b986b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:01.586916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8afdee8-e133-4605-9860-3c06674b986b admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:01.586916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8afdee8-e133-4605-9860-3c06674b986b admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:01.586916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8afdee8-e133-4605-9860-3c06674b986b admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:01.586916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8afdee8-e133-4605-9860-3c06674b986b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:01.587666 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8afdee8-e133-4605-9860-3c06674b986b admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:01.694629 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1345/2663] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:48:01 2022] POST /identity/v3/projects => generated 373 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:48:01.698569 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806817a6-1811-41ff-bab8-0b882e5ba8b9 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:01.709917 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806817a6-1811-41ff-bab8-0b882e5ba8b9 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:01.770474 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702c0037-eba8-4e23-be10-7352b599dbef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:01.771164 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702c0037-eba8-4e23-be10-7352b599dbef admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:01.771458 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702c0037-eba8-4e23-be10-7352b599dbef admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:01.771773 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702c0037-eba8-4e23-be10-7352b599dbef admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:01.772485 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702c0037-eba8-4e23-be10-7352b599dbef admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:01.774498 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702c0037-eba8-4e23-be10-7352b599dbef admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:01.858554 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1319/2664] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:48:01 2022] POST /identity/v3/users => generated 293 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:48:01.863062 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8afdee8-e133-4605-9860-3c06674b986b admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:01.876675 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8afdee8-e133-4605-9860-3c06674b986b admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:01.881616 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccc2b92-6605-459c-ab31-0d90e0a1211d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:01.882414 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccc2b92-6605-459c-ab31-0d90e0a1211d admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:01.882803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccc2b92-6605-459c-ab31-0d90e0a1211d admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:01.883179 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ccc2b92-6605-459c-ab31-0d90e0a1211d admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:01.884051 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ccc2b92-6605-459c-ab31-0d90e0a1211d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:01.886091 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ccc2b92-6605-459c-ab31-0d90e0a1211d admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:01.897781 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1346/2665] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:48:01 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:48:01.902124 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702c0037-eba8-4e23-be10-7352b599dbef admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:01.912941 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702c0037-eba8-4e23-be10-7352b599dbef admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:01.917928 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96920886-1eac-4eee-ba10-1046dbfb332e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:01.919091 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96920886-1eac-4eee-ba10-1046dbfb332e admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:01.919496 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96920886-1eac-4eee-ba10-1046dbfb332e admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:01.919824 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96920886-1eac-4eee-ba10-1046dbfb332e admin admin] PATH_INFO: `/v3/projects/874709a32515449487e4501b94f95cf9/users/b31c21e1d5d0450396851b44f086cf76/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:01.937835 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96920886-1eac-4eee-ba10-1046dbfb332e admin admin] RBAC: Authorizing `identity:create_grant(project_id=874709a32515449487e4501b94f95cf9, user_id=b31c21e1d5d0450396851b44f086cf76, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:01.940779 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96920886-1eac-4eee-ba10-1046dbfb332e admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:02.006420 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1320/2666] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:48:01 2022] PUT /identity/v3/projects/874709a32515449487e4501b94f95cf9/users/b31c21e1d5d0450396851b44f086cf76/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:48:02.010837 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccc2b92-6605-459c-ab31-0d90e0a1211d admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:02.021878 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ccc2b92-6605-459c-ab31-0d90e0a1211d admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:02.082807 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2e5b61-2948-4bb4-aa41-014d59540b31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:02.083480 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2e5b61-2948-4bb4-aa41-014d59540b31 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:02.083768 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2e5b61-2948-4bb4-aa41-014d59540b31 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:02.084051 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2e5b61-2948-4bb4-aa41-014d59540b31 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:02.084782 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c2e5b61-2948-4bb4-aa41-014d59540b31 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:02.086378 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c2e5b61-2948-4bb4-aa41-014d59540b31 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:02.096477 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1347/2667] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:48:02 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:48:02.100805 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96920886-1eac-4eee-ba10-1046dbfb332e admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:02.112391 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96920886-1eac-4eee-ba10-1046dbfb332e admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:02.117294 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d13db0e-d710-4ac0-801e-8fb24c1640eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:02.118199 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d13db0e-d710-4ac0-801e-8fb24c1640eb admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:02.118524 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d13db0e-d710-4ac0-801e-8fb24c1640eb admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:02.118811 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d13db0e-d710-4ac0-801e-8fb24c1640eb admin admin] PATH_INFO: `/v3/projects/874709a32515449487e4501b94f95cf9/users/b31c21e1d5d0450396851b44f086cf76/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:02.120845 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d13db0e-d710-4ac0-801e-8fb24c1640eb admin admin] RBAC: Authorizing `identity:create_grant(project_id=874709a32515449487e4501b94f95cf9, user_id=b31c21e1d5d0450396851b44f086cf76, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:02.124256 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d13db0e-d710-4ac0-801e-8fb24c1640eb admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:02.134838 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1321/2668] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:48:02 2022] PUT /identity/v3/projects/874709a32515449487e4501b94f95cf9/users/b31c21e1d5d0450396851b44f086cf76/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:48:03.444214 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3715f33-21b3-4383-b869-a71d37b0a483 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:03.444813 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3715f33-21b3-4383-b869-a71d37b0a483 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:03.444813 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3715f33-21b3-4383-b869-a71d37b0a483 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:03.478841 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-e3715f33-21b3-4383-b869-a71d37b0a483 None None] MFA Rules not processed for user `b31c21e1d5d0450396851b44f086cf76`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:48:03.480566 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-e3715f33-21b3-4383-b869-a71d37b0a483 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:03.560909 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1348/2669] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:48:03 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:48:03.573243 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d13db0e-d710-4ac0-801e-8fb24c1640eb admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:03.575142 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-4d13db0e-d710-4ac0-801e-8fb24c1640eb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:03.603277 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:48:03.639430 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d13db0e-d710-4ac0-801e-8fb24c1640eb admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:03.643107 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0853b889-663b-4222-9e5e-e95a819a82c6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:03.643750 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0853b889-663b-4222-9e5e-e95a819a82c6 service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:03.644036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0853b889-663b-4222-9e5e-e95a819a82c6 service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:03.644341 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0853b889-663b-4222-9e5e-e95a819a82c6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:03.836414 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0853b889-663b-4222-9e5e-e95a819a82c6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:03.837904 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0853b889-663b-4222-9e5e-e95a819a82c6 service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:03.847529 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1322/2670] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:48:03 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 275 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:48:05.527403 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3715f33-21b3-4383-b869-a71d37b0a483 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:05.537675 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3715f33-21b3-4383-b869-a71d37b0a483 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:05.541249 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09aab918-ced5-4bee-8a3b-fd783faf18d5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:05.541887 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09aab918-ced5-4bee-8a3b-fd783faf18d5 service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:05.542171 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09aab918-ced5-4bee-8a3b-fd783faf18d5 service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:05.542468 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09aab918-ced5-4bee-8a3b-fd783faf18d5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:05.552387 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09aab918-ced5-4bee-8a3b-fd783faf18d5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:05.553879 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09aab918-ced5-4bee-8a3b-fd783faf18d5 service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:05.566129 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1349/2671] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:48:05 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:48:13.843228 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=39 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:48:15.559548 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=98,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:48:20.211229 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0853b889-663b-4222-9e5e-e95a819a82c6 service cinder] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:20.212718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-0853b889-663b-4222-9e5e-e95a819a82c6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:20.279465 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0853b889-663b-4222-9e5e-e95a819a82c6 service cinder] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:20.284080 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ebba4bc-1f73-486b-9732-674d66422718 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:20.285567 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ebba4bc-1f73-486b-9732-674d66422718 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:20.285852 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ebba4bc-1f73-486b-9732-674d66422718 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:20.286133 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ebba4bc-1f73-486b-9732-674d66422718 admin admin] PATH_INFO: `/v3/users/8810a50afb284cdb9dff1b23d3244375` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:20.287598 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ebba4bc-1f73-486b-9732-674d66422718 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8810a50afb284cdb9dff1b23d3244375)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:20.289590 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ebba4bc-1f73-486b-9732-674d66422718 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:20.529289 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-6ebba4bc-1f73-486b-9732-674d66422718 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8810a50afb284cdb9dff1b23d3244375'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:20.623056 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-6ebba4bc-1f73-486b-9732-674d66422718 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8810a50afb284cdb9dff1b23d3244375'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:20.623683 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-6ebba4bc-1f73-486b-9732-674d66422718 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8810a50afb284cdb9dff1b23d3244375'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:20.633332 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-6ebba4bc-1f73-486b-9732-674d66422718 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8810a50afb284cdb9dff1b23d3244375'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:20.647116 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-6ebba4bc-1f73-486b-9732-674d66422718 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8810a50afb284cdb9dff1b23d3244375'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:20.653456 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1323/2672] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:48:20 2022] DELETE /identity/v3/users/8810a50afb284cdb9dff1b23d3244375 => generated 0 bytes in 443 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:48:20.658234 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e073ef21-bc66-485f-8c94-a948c29c0b0d None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:20.663439 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e073ef21-bc66-485f-8c94-a948c29c0b0d None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:20.663439 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e073ef21-bc66-485f-8c94-a948c29c0b0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:20.677578 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-e073ef21-bc66-485f-8c94-a948c29c0b0d None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:48:20.678873 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-e073ef21-bc66-485f-8c94-a948c29c0b0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:20.742847 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1350/2673] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:48:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:48:20.752458 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ebba4bc-1f73-486b-9732-674d66422718 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:20.754526 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-6ebba4bc-1f73-486b-9732-674d66422718 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:20.815256 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ebba4bc-1f73-486b-9732-674d66422718 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:20.820079 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d057e2-0633-4bf4-b93e-45c89eb7a772 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:20.820750 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d057e2-0633-4bf4-b93e-45c89eb7a772 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:20.821040 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d057e2-0633-4bf4-b93e-45c89eb7a772 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:20.821350 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d057e2-0633-4bf4-b93e-45c89eb7a772 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:20.831766 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d057e2-0633-4bf4-b93e-45c89eb7a772 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:20.833211 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d057e2-0633-4bf4-b93e-45c89eb7a772 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:20.843345 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1324/2674] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:48:20 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:48:21.345623 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e073ef21-bc66-485f-8c94-a948c29c0b0d None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:21.347083 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-e073ef21-bc66-485f-8c94-a948c29c0b0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:21.361896 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e073ef21-bc66-485f-8c94-a948c29c0b0d None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:21.368367 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d810173-9621-4b77-be30-dd4784f2d469 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:21.369217 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d810173-9621-4b77-be30-dd4784f2d469 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:21.369600 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d810173-9621-4b77-be30-dd4784f2d469 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:21.369788 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d810173-9621-4b77-be30-dd4784f2d469 admin admin] PATH_INFO: `/v3/projects/3594d8c4a6bc4fdfb378c509820837ac` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:21.371032 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d810173-9621-4b77-be30-dd4784f2d469 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3594d8c4a6bc4fdfb378c509820837ac)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:21.373203 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d810173-9621-4b77-be30-dd4784f2d469 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:21.500907 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-3d810173-9621-4b77-be30-dd4784f2d469 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3594d8c4a6bc4fdfb378c509820837ac', 'request_id': 'req-3d810173-9621-4b77-be30-dd4784f2d469'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:21.507572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-3d810173-9621-4b77-be30-dd4784f2d469 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3594d8c4a6bc4fdfb378c509820837ac', 'request_id': 'req-3d810173-9621-4b77-be30-dd4784f2d469'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:21.590378 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-3d810173-9621-4b77-be30-dd4784f2d469 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3594d8c4a6bc4fdfb378c509820837ac', 'request_id': 'req-3d810173-9621-4b77-be30-dd4784f2d469'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:21.603898 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-3d810173-9621-4b77-be30-dd4784f2d469 admin admin] The token cache is being invalidate because project 3594d8c4a6bc4fdfb378c509820837ac was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:48:21.604191 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-3d810173-9621-4b77-be30-dd4784f2d469 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:21.604848 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-3d810173-9621-4b77-be30-dd4784f2d469 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:21.606649 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1351/2675] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:48:21 2022] DELETE /identity/v3/projects/3594d8c4a6bc4fdfb378c509820837ac => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:48:21.611853 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b4e099-5b83-4762-a6a3-9ae5077ba553 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:21.612339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b4e099-5b83-4762-a6a3-9ae5077ba553 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:21.612339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b4e099-5b83-4762-a6a3-9ae5077ba553 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:21.633078 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-13b4e099-5b83-4762-a6a3-9ae5077ba553 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:48:21.634363 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-13b4e099-5b83-4762-a6a3-9ae5077ba553 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:21.654012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:48:21.701094 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1325/2676] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:48:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:48:21.709559 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1cb96a-0c05-4ac9-9564-4f5c744426f4 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:21.710066 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1cb96a-0c05-4ac9-9564-4f5c744426f4 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:21.710066 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1cb96a-0c05-4ac9-9564-4f5c744426f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:21.729841 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-db1cb96a-0c05-4ac9-9564-4f5c744426f4 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:48:21.731158 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-db1cb96a-0c05-4ac9-9564-4f5c744426f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:21.739683 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1352/2677] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:48:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:48:21.744029 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b4e099-5b83-4762-a6a3-9ae5077ba553 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:21.753996 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b4e099-5b83-4762-a6a3-9ae5077ba553 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:21.758799 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633b7ebd-6f19-4ed8-9927-bdd9e0373a07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:21.759450 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633b7ebd-6f19-4ed8-9927-bdd9e0373a07 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:21.759739 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633b7ebd-6f19-4ed8-9927-bdd9e0373a07 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:21.760017 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633b7ebd-6f19-4ed8-9927-bdd9e0373a07 admin admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:21.760741 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-633b7ebd-6f19-4ed8-9927-bdd9e0373a07 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:48:21.761049 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-633b7ebd-6f19-4ed8-9927-bdd9e0373a07 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:21.762665 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-633b7ebd-6f19-4ed8-9927-bdd9e0373a07 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:21.776484 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1326/2678] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:48:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:48:21.781080 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1cb96a-0c05-4ac9-9564-4f5c744426f4 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:21.791272 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1cb96a-0c05-4ac9-9564-4f5c744426f4 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:21.796582 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa34aad-4447-484b-b68a-f908910e2ad9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:21.798154 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa34aad-4447-484b-b68a-f908910e2ad9 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:21.798154 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa34aad-4447-484b-b68a-f908910e2ad9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:21.798154 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa34aad-4447-484b-b68a-f908910e2ad9 admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:21.798820 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa34aad-4447-484b-b68a-f908910e2ad9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:21.800977 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa34aad-4447-484b-b68a-f908910e2ad9 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:21.806939 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633b7ebd-6f19-4ed8-9927-bdd9e0373a07 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:21.808925 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-633b7ebd-6f19-4ed8-9927-bdd9e0373a07 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:21.825740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633b7ebd-6f19-4ed8-9927-bdd9e0373a07 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:21.830587 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f165cf-19f5-4698-8cbe-a0697e615455 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:21.831455 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f165cf-19f5-4698-8cbe-a0697e615455 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:21.832340 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f165cf-19f5-4698-8cbe-a0697e615455 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:21.832835 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f165cf-19f5-4698-8cbe-a0697e615455 admin admin] PATH_INFO: `/v3/users/53deec137af84c3fbba5d6807a5ca208` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:21.834038 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f165cf-19f5-4698-8cbe-a0697e615455 admin admin] RBAC: Authorizing `identity:delete_user(user_id=53deec137af84c3fbba5d6807a5ca208)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:21.836006 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f165cf-19f5-4698-8cbe-a0697e615455 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:21.894801 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1353/2679] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:48:21 2022] POST /identity/v3/projects => generated 373 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:48:21.899021 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa34aad-4447-484b-b68a-f908910e2ad9 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:21.958010 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:48:21.962964 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa34aad-4447-484b-b68a-f908910e2ad9 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:21.967795 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a067880-cc00-44ec-887c-3fb4f5fec4a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:21.968463 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a067880-cc00-44ec-887c-3fb4f5fec4a5 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:21.968754 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a067880-cc00-44ec-887c-3fb4f5fec4a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:21.969038 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a067880-cc00-44ec-887c-3fb4f5fec4a5 admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:21.969724 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a067880-cc00-44ec-887c-3fb4f5fec4a5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:21.971736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a067880-cc00-44ec-887c-3fb4f5fec4a5 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:22.027465 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-a6f165cf-19f5-4698-8cbe-a0697e615455 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '53deec137af84c3fbba5d6807a5ca208'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:22.041482 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1354/2680] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:48:21 2022] POST /identity/v3/users => generated 287 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:48:22.045434 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a067880-cc00-44ec-887c-3fb4f5fec4a5 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:22.055475 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a067880-cc00-44ec-887c-3fb4f5fec4a5 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:22.060419 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce96a072-7cc5-4ee3-ab5f-4f0daa3d3380 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:22.061014 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce96a072-7cc5-4ee3-ab5f-4f0daa3d3380 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:22.061304 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce96a072-7cc5-4ee3-ab5f-4f0daa3d3380 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:22.061587 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce96a072-7cc5-4ee3-ab5f-4f0daa3d3380 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:22.062399 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce96a072-7cc5-4ee3-ab5f-4f0daa3d3380 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:22.073186 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce96a072-7cc5-4ee3-ab5f-4f0daa3d3380 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:22.075673 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1355/2681] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:48:22 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 12 17:48:22.079512 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce96a072-7cc5-4ee3-ab5f-4f0daa3d3380 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:22.091412 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce96a072-7cc5-4ee3-ab5f-4f0daa3d3380 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:22.096391 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5580c5b7-5d7f-4fbe-949c-ce825baca5d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:22.096834 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5580c5b7-5d7f-4fbe-949c-ce825baca5d4 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:22.097127 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5580c5b7-5d7f-4fbe-949c-ce825baca5d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:22.097665 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5580c5b7-5d7f-4fbe-949c-ce825baca5d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:22.098241 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5580c5b7-5d7f-4fbe-949c-ce825baca5d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:22.099861 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5580c5b7-5d7f-4fbe-949c-ce825baca5d4 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:22.110456 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1356/2682] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:48:22 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 12 17:48:22.114371 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5580c5b7-5d7f-4fbe-949c-ce825baca5d4 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:22.115840 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-a6f165cf-19f5-4698-8cbe-a0697e615455 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '53deec137af84c3fbba5d6807a5ca208'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:22.116689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-a6f165cf-19f5-4698-8cbe-a0697e615455 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '53deec137af84c3fbba5d6807a5ca208'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:22.125367 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5580c5b7-5d7f-4fbe-949c-ce825baca5d4 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:22.125968 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-a6f165cf-19f5-4698-8cbe-a0697e615455 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '53deec137af84c3fbba5d6807a5ca208'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:22.131466 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77bdaad8-3314-4a97-be82-415e3b56ff36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:22.132339 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bdaad8-3314-4a97-be82-415e3b56ff36 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:22.132632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bdaad8-3314-4a97-be82-415e3b56ff36 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:22.132934 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77bdaad8-3314-4a97-be82-415e3b56ff36 admin admin] PATH_INFO: `/v3/projects/fbcca74cfd4a462fbb6ba5143c68a460/users/bb203b8551614904a5a85d6260f894fb/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:22.144906 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-a6f165cf-19f5-4698-8cbe-a0697e615455 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '53deec137af84c3fbba5d6807a5ca208'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:22.150318 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77bdaad8-3314-4a97-be82-415e3b56ff36 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fbcca74cfd4a462fbb6ba5143c68a460, user_id=bb203b8551614904a5a85d6260f894fb, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:22.151623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1327/2683] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:48:21 2022] DELETE /identity/v3/users/53deec137af84c3fbba5d6807a5ca208 => generated 0 bytes in 345 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:48:22.153882 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77bdaad8-3314-4a97-be82-415e3b56ff36 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:22.262704 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1357/2684] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:48:22 2022] PUT /identity/v3/projects/fbcca74cfd4a462fbb6ba5143c68a460/users/bb203b8551614904a5a85d6260f894fb/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:48:22.314115 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713cdc47-8aac-473a-aa1e-6d25ecc93681 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:22.314541 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713cdc47-8aac-473a-aa1e-6d25ecc93681 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:22.314745 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713cdc47-8aac-473a-aa1e-6d25ecc93681 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:22.343793 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-713cdc47-8aac-473a-aa1e-6d25ecc93681 None None] MFA Rules not processed for user `bb203b8551614904a5a85d6260f894fb`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:48:22.345026 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-713cdc47-8aac-473a-aa1e-6d25ecc93681 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:22.420682 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1328/2685] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:48:22 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:48:22.432215 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77bdaad8-3314-4a97-be82-415e3b56ff36 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:22.434064 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-77bdaad8-3314-4a97-be82-415e3b56ff36 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:22.472294 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713cdc47-8aac-473a-aa1e-6d25ecc93681 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:22.476210 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-713cdc47-8aac-473a-aa1e-6d25ecc93681 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:22.502928 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:48:22.524739 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77bdaad8-3314-4a97-be82-415e3b56ff36 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:22.529848 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0501a85-b570-4282-bf2f-a7ed59c76977 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:22.530544 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0501a85-b570-4282-bf2f-a7ed59c76977 service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:22.530856 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0501a85-b570-4282-bf2f-a7ed59c76977 service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:22.531281 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0501a85-b570-4282-bf2f-a7ed59c76977 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:22.540772 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0501a85-b570-4282-bf2f-a7ed59c76977 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:22.542296 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0501a85-b570-4282-bf2f-a7ed59c76977 service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:22.558211 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1358/2686] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:48:22 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:48:22.569148 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713cdc47-8aac-473a-aa1e-6d25ecc93681 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:22.574418 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7c3703-e7cb-497d-832b-38dc12d27151 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:22.575673 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7c3703-e7cb-497d-832b-38dc12d27151 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:22.575806 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7c3703-e7cb-497d-832b-38dc12d27151 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:22.576073 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7c3703-e7cb-497d-832b-38dc12d27151 admin admin] PATH_INFO: `/v3/projects/dcfce527d1fa46f398920d05eab9ae78` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:22.577360 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce7c3703-e7cb-497d-832b-38dc12d27151 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dcfce527d1fa46f398920d05eab9ae78)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:22.579357 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce7c3703-e7cb-497d-832b-38dc12d27151 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:22.702374 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-ce7c3703-e7cb-497d-832b-38dc12d27151 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dcfce527d1fa46f398920d05eab9ae78', 'request_id': 'req-ce7c3703-e7cb-497d-832b-38dc12d27151'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:22.708792 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-ce7c3703-e7cb-497d-832b-38dc12d27151 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dcfce527d1fa46f398920d05eab9ae78', 'request_id': 'req-ce7c3703-e7cb-497d-832b-38dc12d27151'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:22.714628 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-ce7c3703-e7cb-497d-832b-38dc12d27151 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dcfce527d1fa46f398920d05eab9ae78', 'request_id': 'req-ce7c3703-e7cb-497d-832b-38dc12d27151'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:22.797051 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-ce7c3703-e7cb-497d-832b-38dc12d27151 admin admin] The token cache is being invalidate because project dcfce527d1fa46f398920d05eab9ae78 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:48:22.797653 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-ce7c3703-e7cb-497d-832b-38dc12d27151 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:22.798579 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-ce7c3703-e7cb-497d-832b-38dc12d27151 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:22.801347 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1329/2687] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:48:22 2022] DELETE /identity/v3/projects/dcfce527d1fa46f398920d05eab9ae78 => generated 0 bytes in 331 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:48:22.805389 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0501a85-b570-4282-bf2f-a7ed59c76977 service cinder] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:22.817547 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0501a85-b570-4282-bf2f-a7ed59c76977 service cinder] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:22.887355 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f65fbb9-ee24-447d-98ee-c7669f8ccb74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:22.888258 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f65fbb9-ee24-447d-98ee-c7669f8ccb74 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:22.888545 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f65fbb9-ee24-447d-98ee-c7669f8ccb74 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:22.888824 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f65fbb9-ee24-447d-98ee-c7669f8ccb74 admin admin] PATH_INFO: `/v3/users/b31c21e1d5d0450396851b44f086cf76` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:22.890108 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f65fbb9-ee24-447d-98ee-c7669f8ccb74 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b31c21e1d5d0450396851b44f086cf76)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:22.892080 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f65fbb9-ee24-447d-98ee-c7669f8ccb74 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:23.096450 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=90,keystone:INSERT=4,keystone:DELETE=6 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:48:23.100530 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-9f65fbb9-ee24-447d-98ee-c7669f8ccb74 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b31c21e1d5d0450396851b44f086cf76'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:23.101448 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-9f65fbb9-ee24-447d-98ee-c7669f8ccb74 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b31c21e1d5d0450396851b44f086cf76'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:23.102035 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-9f65fbb9-ee24-447d-98ee-c7669f8ccb74 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b31c21e1d5d0450396851b44f086cf76'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:23.120652 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-9f65fbb9-ee24-447d-98ee-c7669f8ccb74 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b31c21e1d5d0450396851b44f086cf76'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:23.143124 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-9f65fbb9-ee24-447d-98ee-c7669f8ccb74 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b31c21e1d5d0450396851b44f086cf76'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:23.264003 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1359/2688] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:48:22 2022] DELETE /identity/v3/users/b31c21e1d5d0450396851b44f086cf76 => generated 0 bytes in 459 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:48:23.598185 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7c3703-e7cb-497d-832b-38dc12d27151 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:23.599645 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-ce7c3703-e7cb-497d-832b-38dc12d27151 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:23.625640 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7c3703-e7cb-497d-832b-38dc12d27151 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:23.698455 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18bdd27-b5ee-4e0d-8b9f-e734eaafb868 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:23.699333 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18bdd27-b5ee-4e0d-8b9f-e734eaafb868 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:23.699619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18bdd27-b5ee-4e0d-8b9f-e734eaafb868 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:23.699906 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18bdd27-b5ee-4e0d-8b9f-e734eaafb868 admin admin] PATH_INFO: `/v3/projects/874709a32515449487e4501b94f95cf9` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:23.701124 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d18bdd27-b5ee-4e0d-8b9f-e734eaafb868 admin admin] RBAC: Authorizing `identity:delete_project(project_id=874709a32515449487e4501b94f95cf9)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:23.703070 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d18bdd27-b5ee-4e0d-8b9f-e734eaafb868 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:23.848992 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:48:23.854077 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-d18bdd27-b5ee-4e0d-8b9f-e734eaafb868 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '874709a32515449487e4501b94f95cf9', 'request_id': 'req-d18bdd27-b5ee-4e0d-8b9f-e734eaafb868'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:23.861112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-d18bdd27-b5ee-4e0d-8b9f-e734eaafb868 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '874709a32515449487e4501b94f95cf9', 'request_id': 'req-d18bdd27-b5ee-4e0d-8b9f-e734eaafb868'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:23.868196 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-d18bdd27-b5ee-4e0d-8b9f-e734eaafb868 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '874709a32515449487e4501b94f95cf9', 'request_id': 'req-d18bdd27-b5ee-4e0d-8b9f-e734eaafb868'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:23.949911 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-d18bdd27-b5ee-4e0d-8b9f-e734eaafb868 admin admin] The token cache is being invalidate because project 874709a32515449487e4501b94f95cf9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:48:23.950209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-d18bdd27-b5ee-4e0d-8b9f-e734eaafb868 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:23.951016 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-d18bdd27-b5ee-4e0d-8b9f-e734eaafb868 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:23.958320 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1330/2689] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:48:23 2022] DELETE /identity/v3/projects/874709a32515449487e4501b94f95cf9 => generated 0 bytes in 356 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:48:23.958945 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5feb5911-5a60-463c-9501-7153ec15f480 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:23.958945 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5feb5911-5a60-463c-9501-7153ec15f480 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:23.958945 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5feb5911-5a60-463c-9501-7153ec15f480 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:23.978921 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-5feb5911-5a60-463c-9501-7153ec15f480 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:48:23.980269 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-5feb5911-5a60-463c-9501-7153ec15f480 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:24.054642 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1360/2690] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:48:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:48:24.059839 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfce756-6a48-4139-a312-f7db4c39a343 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:24.060137 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfce756-6a48-4139-a312-f7db4c39a343 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:24.060426 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dfce756-6a48-4139-a312-f7db4c39a343 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:24.080307 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-3dfce756-6a48-4139-a312-f7db4c39a343 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:48:24.082008 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-3dfce756-6a48-4139-a312-f7db4c39a343 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:24.091689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1331/2691] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:48:24 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:48:24.096242 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5feb5911-5a60-463c-9501-7153ec15f480 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:24.107081 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5feb5911-5a60-463c-9501-7153ec15f480 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:24.111912 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e1bf97-e159-4535-86c1-b23ed90ade20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:24.112589 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e1bf97-e159-4535-86c1-b23ed90ade20 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:24.113573 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e1bf97-e159-4535-86c1-b23ed90ade20 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:24.114131 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79e1bf97-e159-4535-86c1-b23ed90ade20 admin admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:24.114926 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e1bf97-e159-4535-86c1-b23ed90ade20 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:48:24.115237 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e1bf97-e159-4535-86c1-b23ed90ade20 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:24.116854 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79e1bf97-e159-4535-86c1-b23ed90ade20 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:24.129792 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1361/2692] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:48:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:48:24.134400 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfce756-6a48-4139-a312-f7db4c39a343 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:24.144207 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dfce756-6a48-4139-a312-f7db4c39a343 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:24.149229 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24cd152c-5c45-4142-94ef-e928fb243851 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:24.149913 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cd152c-5c45-4142-94ef-e928fb243851 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:24.150205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cd152c-5c45-4142-94ef-e928fb243851 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:24.150519 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cd152c-5c45-4142-94ef-e928fb243851 admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:24.151246 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24cd152c-5c45-4142-94ef-e928fb243851 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:24.153196 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24cd152c-5c45-4142-94ef-e928fb243851 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:24.266512 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1332/2693] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:48:24 2022] POST /identity/v3/projects => generated 385 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:48:24.270698 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e1bf97-e159-4535-86c1-b23ed90ade20 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:24.280955 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79e1bf97-e159-4535-86c1-b23ed90ade20 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:24.343359 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6368c9eb-031d-44bc-a8d9-788a4e6d037b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:24.344032 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6368c9eb-031d-44bc-a8d9-788a4e6d037b admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:24.344321 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6368c9eb-031d-44bc-a8d9-788a4e6d037b admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:24.344614 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6368c9eb-031d-44bc-a8d9-788a4e6d037b admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:24.345301 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6368c9eb-031d-44bc-a8d9-788a4e6d037b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:24.347902 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6368c9eb-031d-44bc-a8d9-788a4e6d037b admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:24.434162 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1362/2694] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:48:24 2022] POST /identity/v3/users => generated 293 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:48:24.441131 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24cd152c-5c45-4142-94ef-e928fb243851 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:24.457091 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24cd152c-5c45-4142-94ef-e928fb243851 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:24.459127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4d53c8-6b59-42e0-b7ff-e5c4709462aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:24.459806 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4d53c8-6b59-42e0-b7ff-e5c4709462aa admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:24.460111 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4d53c8-6b59-42e0-b7ff-e5c4709462aa admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:24.460407 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4d53c8-6b59-42e0-b7ff-e5c4709462aa admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:24.461120 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b4d53c8-6b59-42e0-b7ff-e5c4709462aa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:24.462808 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b4d53c8-6b59-42e0-b7ff-e5c4709462aa admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:24.474619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1333/2695] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:48:24 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:48:24.478711 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6368c9eb-031d-44bc-a8d9-788a4e6d037b admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:24.489786 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6368c9eb-031d-44bc-a8d9-788a4e6d037b admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:24.494593 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee83712-206d-431f-893c-0a8a95a52d96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:24.495266 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee83712-206d-431f-893c-0a8a95a52d96 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:24.495563 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee83712-206d-431f-893c-0a8a95a52d96 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:24.495838 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee83712-206d-431f-893c-0a8a95a52d96 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:24.496581 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee83712-206d-431f-893c-0a8a95a52d96 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:24.498619 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee83712-206d-431f-893c-0a8a95a52d96 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:24.509948 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1363/2696] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:48:24 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:48:24.516994 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4d53c8-6b59-42e0-b7ff-e5c4709462aa admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:24.527796 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4d53c8-6b59-42e0-b7ff-e5c4709462aa admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:24.532506 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10bfaebb-b6c5-4193-970a-cd00c68afcda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:24.533853 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bfaebb-b6c5-4193-970a-cd00c68afcda admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:24.535512 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bfaebb-b6c5-4193-970a-cd00c68afcda admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:24.536024 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bfaebb-b6c5-4193-970a-cd00c68afcda admin admin] PATH_INFO: `/v3/projects/fa7e2c910ccf4fa3b94cd780a0011ef8/users/756d496daf0f4428b2157d6afdff2e80/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:24.552856 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10bfaebb-b6c5-4193-970a-cd00c68afcda admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa7e2c910ccf4fa3b94cd780a0011ef8, user_id=756d496daf0f4428b2157d6afdff2e80, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:24.555649 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10bfaebb-b6c5-4193-970a-cd00c68afcda admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:24.646959 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1334/2697] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:48:24 2022] PUT /identity/v3/projects/fa7e2c910ccf4fa3b94cd780a0011ef8/users/756d496daf0f4428b2157d6afdff2e80/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:48:24.658189 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfcd4d9-30af-4ff3-9da4-25f8059a11c6 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:24.658655 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfcd4d9-30af-4ff3-9da4-25f8059a11c6 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:24.658952 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfcd4d9-30af-4ff3-9da4-25f8059a11c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:24.687744 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-9cfcd4d9-30af-4ff3-9da4-25f8059a11c6 None None] MFA Rules not processed for user `756d496daf0f4428b2157d6afdff2e80`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:48:24.688970 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-9cfcd4d9-30af-4ff3-9da4-25f8059a11c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:24.702253 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:DELETE=4,keystone:SELECT=92,keystone:INSERT=4 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:48:24.755927 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1364/2698] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:48:24 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:48:24.760469 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10bfaebb-b6c5-4193-970a-cd00c68afcda admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:24.836388 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10bfaebb-b6c5-4193-970a-cd00c68afcda admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:24.841161 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64778bcf-3acc-48c5-aa70-043753a35f19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:24.841832 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64778bcf-3acc-48c5-aa70-043753a35f19 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:24.842189 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64778bcf-3acc-48c5-aa70-043753a35f19 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:24.842567 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64778bcf-3acc-48c5-aa70-043753a35f19 admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:24.850662 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64778bcf-3acc-48c5-aa70-043753a35f19 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:24.850662 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64778bcf-3acc-48c5-aa70-043753a35f19 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:24.928815 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1335/2699] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:48:24 2022] POST /identity/v3/projects => generated 385 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:48:24.934481 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfcd4d9-30af-4ff3-9da4-25f8059a11c6 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:24.944406 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfcd4d9-30af-4ff3-9da4-25f8059a11c6 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:25.000959 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e190266c-fc5f-4f8d-a95b-aaef7b0ff360 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:25.001681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e190266c-fc5f-4f8d-a95b-aaef7b0ff360 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:25.001899 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e190266c-fc5f-4f8d-a95b-aaef7b0ff360 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:25.002189 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e190266c-fc5f-4f8d-a95b-aaef7b0ff360 admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:25.002912 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e190266c-fc5f-4f8d-a95b-aaef7b0ff360 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:25.005592 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e190266c-fc5f-4f8d-a95b-aaef7b0ff360 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:25.112279 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1365/2700] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:48:24 2022] POST /identity/v3/users => generated 299 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:48:25.113580 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64778bcf-3acc-48c5-aa70-043753a35f19 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:25.124441 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64778bcf-3acc-48c5-aa70-043753a35f19 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:25.130703 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a35cf4-1b31-4251-81aa-e668230975c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:25.131564 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a35cf4-1b31-4251-81aa-e668230975c0 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:25.132580 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a35cf4-1b31-4251-81aa-e668230975c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:25.132886 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a35cf4-1b31-4251-81aa-e668230975c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:25.135730 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a35cf4-1b31-4251-81aa-e668230975c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:25.135995 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a35cf4-1b31-4251-81aa-e668230975c0 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:25.150884 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1336/2701] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:48:25 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:48:25.155530 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e190266c-fc5f-4f8d-a95b-aaef7b0ff360 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:25.166835 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e190266c-fc5f-4f8d-a95b-aaef7b0ff360 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:25.172201 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-789cd6b5-eb6e-4924-841e-8f1c40644179 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:25.173176 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789cd6b5-eb6e-4924-841e-8f1c40644179 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:25.173554 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789cd6b5-eb6e-4924-841e-8f1c40644179 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:25.174271 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789cd6b5-eb6e-4924-841e-8f1c40644179 admin admin] PATH_INFO: `/v3/projects/299dcb61892d4e9a8b2c023bc157c37c/users/6747fe9912ab4e4791b81047650b9b3b/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:25.195582 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-789cd6b5-eb6e-4924-841e-8f1c40644179 admin admin] RBAC: Authorizing `identity:create_grant(project_id=299dcb61892d4e9a8b2c023bc157c37c, user_id=6747fe9912ab4e4791b81047650b9b3b, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:25.198406 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-789cd6b5-eb6e-4924-841e-8f1c40644179 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:25.284989 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1366/2702] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:48:25 2022] PUT /identity/v3/projects/299dcb61892d4e9a8b2c023bc157c37c/users/6747fe9912ab4e4791b81047650b9b3b/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:48:25.287196 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a35cf4-1b31-4251-81aa-e668230975c0 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:25.297870 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a35cf4-1b31-4251-81aa-e668230975c0 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:25.339952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:48:25.356667 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af4b2a2-1d02-4bcf-a2c2-e9b4a7191487 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:25.357339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af4b2a2-1d02-4bcf-a2c2-e9b4a7191487 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:25.357626 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af4b2a2-1d02-4bcf-a2c2-e9b4a7191487 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:25.357906 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af4b2a2-1d02-4bcf-a2c2-e9b4a7191487 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:25.358667 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af4b2a2-1d02-4bcf-a2c2-e9b4a7191487 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:25.360244 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af4b2a2-1d02-4bcf-a2c2-e9b4a7191487 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:25.374626 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1337/2703] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:48:25 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:48:25.376298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-789cd6b5-eb6e-4924-841e-8f1c40644179 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:25.387929 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-789cd6b5-eb6e-4924-841e-8f1c40644179 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:25.393798 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e9f41a-c1ef-4654-bfcb-8046ab59f236 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:25.394948 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e9f41a-c1ef-4654-bfcb-8046ab59f236 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:25.395241 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e9f41a-c1ef-4654-bfcb-8046ab59f236 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:25.395528 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e9f41a-c1ef-4654-bfcb-8046ab59f236 admin admin] PATH_INFO: `/v3/projects/299dcb61892d4e9a8b2c023bc157c37c/users/6747fe9912ab4e4791b81047650b9b3b/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:25.397557 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e9f41a-c1ef-4654-bfcb-8046ab59f236 admin admin] RBAC: Authorizing `identity:create_grant(project_id=299dcb61892d4e9a8b2c023bc157c37c, user_id=6747fe9912ab4e4791b81047650b9b3b, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:25.400371 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e9f41a-c1ef-4654-bfcb-8046ab59f236 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:25.410961 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1367/2704] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:48:25 2022] PUT /identity/v3/projects/299dcb61892d4e9a8b2c023bc157c37c/users/6747fe9912ab4e4791b81047650b9b3b/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:48:25.421372 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893a188e-ee61-4c6c-9ab0-4da9535c9cab None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:25.421670 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893a188e-ee61-4c6c-9ab0-4da9535c9cab None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:25.422720 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893a188e-ee61-4c6c-9ab0-4da9535c9cab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:25.449470 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-893a188e-ee61-4c6c-9ab0-4da9535c9cab None None] MFA Rules not processed for user `6747fe9912ab4e4791b81047650b9b3b`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:48:25.450710 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-893a188e-ee61-4c6c-9ab0-4da9535c9cab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:25.520947 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1338/2705] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:48:25 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:48:25.533795 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e9f41a-c1ef-4654-bfcb-8046ab59f236 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:25.535165 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-b2e9f41a-c1ef-4654-bfcb-8046ab59f236 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:25.614522 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e9f41a-c1ef-4654-bfcb-8046ab59f236 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:25.618336 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-734d0331-fa40-481a-bf3a-e88020ad95f1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:25.619042 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734d0331-fa40-481a-bf3a-e88020ad95f1 service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:25.619333 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734d0331-fa40-481a-bf3a-e88020ad95f1 service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:25.619621 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734d0331-fa40-481a-bf3a-e88020ad95f1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:25.648095 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:48:25.683048 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-734d0331-fa40-481a-bf3a-e88020ad95f1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:25.684492 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-734d0331-fa40-481a-bf3a-e88020ad95f1 service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:25.694085 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1368/2706] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:48:25 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:48:28.216251 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ba0b46-06cb-46ca-bd9d-75ba5fc6dc80 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:28.216731 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ba0b46-06cb-46ca-bd9d-75ba5fc6dc80 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:28.217126 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ba0b46-06cb-46ca-bd9d-75ba5fc6dc80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:28.235741 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-75ba0b46-06cb-46ca-bd9d-75ba5fc6dc80 None None] MFA Rules not processed for user `e3d86a1cde9f407cae8ac050fe000d47`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:48:28.237222 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-75ba0b46-06cb-46ca-bd9d-75ba5fc6dc80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:28.301481 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1339/2707] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:48:28 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:48:30.902014 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-734d0331-fa40-481a-bf3a-e88020ad95f1 service cinder] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:30.912288 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-734d0331-fa40-481a-bf3a-e88020ad95f1 service cinder] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:30.915978 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-530d4d12-feed-4990-8e8d-6ce19dbb17d2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:30.916630 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530d4d12-feed-4990-8e8d-6ce19dbb17d2 service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:30.916919 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530d4d12-feed-4990-8e8d-6ce19dbb17d2 service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:30.917217 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530d4d12-feed-4990-8e8d-6ce19dbb17d2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:30.928293 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-530d4d12-feed-4990-8e8d-6ce19dbb17d2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:30.929759 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-530d4d12-feed-4990-8e8d-6ce19dbb17d2 service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:30.939958 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1369/2708] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:48:30 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:48:34.163796 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ba0b46-06cb-46ca-bd9d-75ba5fc6dc80 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:34.169300 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-75ba0b46-06cb-46ca-bd9d-75ba5fc6dc80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:34.230255 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ba0b46-06cb-46ca-bd9d-75ba5fc6dc80 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:34.235014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6164e41-5bbb-4ac3-ac87-d98a6e930287 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:34.235904 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6164e41-5bbb-4ac3-ac87-d98a6e930287 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:34.236192 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6164e41-5bbb-4ac3-ac87-d98a6e930287 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:34.236474 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6164e41-5bbb-4ac3-ac87-d98a6e930287 admin admin] PATH_INFO: `/v3/users/bb203b8551614904a5a85d6260f894fb` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:34.237702 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6164e41-5bbb-4ac3-ac87-d98a6e930287 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bb203b8551614904a5a85d6260f894fb)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:34.240289 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6164e41-5bbb-4ac3-ac87-d98a6e930287 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:34.251487 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:48:34.494674 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-a6164e41-5bbb-4ac3-ac87-d98a6e930287 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bb203b8551614904a5a85d6260f894fb'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:34.634565 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-a6164e41-5bbb-4ac3-ac87-d98a6e930287 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bb203b8551614904a5a85d6260f894fb'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:34.635473 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-a6164e41-5bbb-4ac3-ac87-d98a6e930287 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bb203b8551614904a5a85d6260f894fb'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:34.645094 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-a6164e41-5bbb-4ac3-ac87-d98a6e930287 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bb203b8551614904a5a85d6260f894fb'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:34.662300 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-a6164e41-5bbb-4ac3-ac87-d98a6e930287 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bb203b8551614904a5a85d6260f894fb'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:34.668748 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1340/2709] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:48:34 2022] DELETE /identity/v3/users/bb203b8551614904a5a85d6260f894fb => generated 0 bytes in 505 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:48:34.673404 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e1d86d-e86d-42d1-8220-710faac903d8 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:34.673892 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e1d86d-e86d-42d1-8220-710faac903d8 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:34.674014 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e1d86d-e86d-42d1-8220-710faac903d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:34.695391 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-d9e1d86d-e86d-42d1-8220-710faac903d8 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:48:34.696626 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-d9e1d86d-e86d-42d1-8220-710faac903d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:34.760865 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1370/2710] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:48:34 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:48:34.771964 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6164e41-5bbb-4ac3-ac87-d98a6e930287 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:34.774132 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-a6164e41-5bbb-4ac3-ac87-d98a6e930287 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:34.839700 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6164e41-5bbb-4ac3-ac87-d98a6e930287 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:34.843267 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ae611e-5ed3-4236-b1c8-13f205d4d940 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:34.843917 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ae611e-5ed3-4236-b1c8-13f205d4d940 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:34.844211 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ae611e-5ed3-4236-b1c8-13f205d4d940 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:34.844493 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ae611e-5ed3-4236-b1c8-13f205d4d940 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:34.853926 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ae611e-5ed3-4236-b1c8-13f205d4d940 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:34.855419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ae611e-5ed3-4236-b1c8-13f205d4d940 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:34.865471 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1341/2711] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:48:34 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:48:35.270019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e1d86d-e86d-42d1-8220-710faac903d8 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:35.272098 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-d9e1d86d-e86d-42d1-8220-710faac903d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:35.285608 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e1d86d-e86d-42d1-8220-710faac903d8 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:35.291406 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab2ac29-a5cc-40d4-a515-cfe644dc1385 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:35.292377 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab2ac29-a5cc-40d4-a515-cfe644dc1385 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:35.292735 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab2ac29-a5cc-40d4-a515-cfe644dc1385 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:35.293380 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab2ac29-a5cc-40d4-a515-cfe644dc1385 admin admin] PATH_INFO: `/v3/projects/fbcca74cfd4a462fbb6ba5143c68a460` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:35.295548 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab2ac29-a5cc-40d4-a515-cfe644dc1385 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fbcca74cfd4a462fbb6ba5143c68a460)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:35.296976 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab2ac29-a5cc-40d4-a515-cfe644dc1385 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:35.405574 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-9ab2ac29-a5cc-40d4-a515-cfe644dc1385 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fbcca74cfd4a462fbb6ba5143c68a460', 'request_id': 'req-9ab2ac29-a5cc-40d4-a515-cfe644dc1385'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:35.410984 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-9ab2ac29-a5cc-40d4-a515-cfe644dc1385 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fbcca74cfd4a462fbb6ba5143c68a460', 'request_id': 'req-9ab2ac29-a5cc-40d4-a515-cfe644dc1385'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:35.479211 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-9ab2ac29-a5cc-40d4-a515-cfe644dc1385 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fbcca74cfd4a462fbb6ba5143c68a460', 'request_id': 'req-9ab2ac29-a5cc-40d4-a515-cfe644dc1385'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:35.491803 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-9ab2ac29-a5cc-40d4-a515-cfe644dc1385 admin admin] The token cache is being invalidate because project fbcca74cfd4a462fbb6ba5143c68a460 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:48:35.495536 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-9ab2ac29-a5cc-40d4-a515-cfe644dc1385 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:35.495536 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-9ab2ac29-a5cc-40d4-a515-cfe644dc1385 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:35.495536 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1371/2712] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:48:35 2022] DELETE /identity/v3/projects/fbcca74cfd4a462fbb6ba5143c68a460 => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:48:35.499230 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e89b49-dfea-4f7d-9201-f245836ae20e None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:35.499684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e89b49-dfea-4f7d-9201-f245836ae20e None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:35.499780 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e89b49-dfea-4f7d-9201-f245836ae20e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:35.517074 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-96e89b49-dfea-4f7d-9201-f245836ae20e None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:48:35.518265 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-96e89b49-dfea-4f7d-9201-f245836ae20e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:35.581269 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1342/2713] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:48:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:48:35.587236 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da95c9cd-9467-428e-a5c9-b6804f28bf31 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:35.587536 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da95c9cd-9467-428e-a5c9-b6804f28bf31 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:35.592052 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da95c9cd-9467-428e-a5c9-b6804f28bf31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:35.607266 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-da95c9cd-9467-428e-a5c9-b6804f28bf31 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:48:35.608484 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-da95c9cd-9467-428e-a5c9-b6804f28bf31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:35.617343 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1372/2714] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:48:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:48:35.621799 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e89b49-dfea-4f7d-9201-f245836ae20e None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:35.631902 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e89b49-dfea-4f7d-9201-f245836ae20e None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:35.636782 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b6350c-a778-4468-8f9b-b0a34e2dd9ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:35.637472 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b6350c-a778-4468-8f9b-b0a34e2dd9ec admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:35.637914 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b6350c-a778-4468-8f9b-b0a34e2dd9ec admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:35.638107 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b6350c-a778-4468-8f9b-b0a34e2dd9ec admin admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:35.638875 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b6350c-a778-4468-8f9b-b0a34e2dd9ec admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:48:35.639192 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b6350c-a778-4468-8f9b-b0a34e2dd9ec admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:35.640780 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b6350c-a778-4468-8f9b-b0a34e2dd9ec admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:35.653595 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1343/2715] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:48:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:48:35.657944 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da95c9cd-9467-428e-a5c9-b6804f28bf31 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:35.667911 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da95c9cd-9467-428e-a5c9-b6804f28bf31 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:35.673122 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa5412b-6217-4246-8277-d8c651c67c27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:35.673527 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa5412b-6217-4246-8277-d8c651c67c27 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:35.674066 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa5412b-6217-4246-8277-d8c651c67c27 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:35.674066 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa5412b-6217-4246-8277-d8c651c67c27 admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:35.674838 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa5412b-6217-4246-8277-d8c651c67c27 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:35.676763 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa5412b-6217-4246-8277-d8c651c67c27 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:35.755679 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1373/2716] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:48:35 2022] POST /identity/v3/projects => generated 381 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:48:35.760010 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b6350c-a778-4468-8f9b-b0a34e2dd9ec admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:35.770062 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b6350c-a778-4468-8f9b-b0a34e2dd9ec admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:35.782932 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:48:35.828297 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6a5662-2d0a-4d85-88a2-dcf1da113a88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:35.829354 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6a5662-2d0a-4d85-88a2-dcf1da113a88 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:35.829584 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6a5662-2d0a-4d85-88a2-dcf1da113a88 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:35.829876 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6a5662-2d0a-4d85-88a2-dcf1da113a88 admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:35.830643 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6a5662-2d0a-4d85-88a2-dcf1da113a88 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:35.832546 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6a5662-2d0a-4d85-88a2-dcf1da113a88 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:35.963756 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1344/2717] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:48:35 2022] POST /identity/v3/users => generated 291 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:48:35.964926 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa5412b-6217-4246-8277-d8c651c67c27 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:35.977558 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa5412b-6217-4246-8277-d8c651c67c27 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:35.979295 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:48:35.983189 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972da2c2-1f74-40b2-80d9-0d97b7eeab0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:35.985344 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972da2c2-1f74-40b2-80d9-0d97b7eeab0a admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:35.985344 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972da2c2-1f74-40b2-80d9-0d97b7eeab0a admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:35.985344 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972da2c2-1f74-40b2-80d9-0d97b7eeab0a admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:35.985344 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972da2c2-1f74-40b2-80d9-0d97b7eeab0a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:35.987128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972da2c2-1f74-40b2-80d9-0d97b7eeab0a admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:35.998258 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1374/2718] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:48:35 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 12 17:48:36.002474 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6a5662-2d0a-4d85-88a2-dcf1da113a88 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:36.013376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6a5662-2d0a-4d85-88a2-dcf1da113a88 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:36.018106 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1260f25c-6641-41d1-b51d-0d0cb722fb25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:36.018802 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1260f25c-6641-41d1-b51d-0d0cb722fb25 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:36.019093 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1260f25c-6641-41d1-b51d-0d0cb722fb25 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:36.019378 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1260f25c-6641-41d1-b51d-0d0cb722fb25 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:36.020109 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1260f25c-6641-41d1-b51d-0d0cb722fb25 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:36.022380 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1260f25c-6641-41d1-b51d-0d0cb722fb25 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:36.033185 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1345/2719] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:48:36 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:48:36.036983 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972da2c2-1f74-40b2-80d9-0d97b7eeab0a admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:36.047045 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972da2c2-1f74-40b2-80d9-0d97b7eeab0a admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:36.051565 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83efaee-f06e-4eaf-b41d-679c940a53bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:36.052632 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83efaee-f06e-4eaf-b41d-679c940a53bf admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:36.053016 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83efaee-f06e-4eaf-b41d-679c940a53bf admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:36.053297 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83efaee-f06e-4eaf-b41d-679c940a53bf admin admin] PATH_INFO: `/v3/projects/c3b0319262bc468d82fa3fba94540142/users/01d3c959a87247cd833db555fdef9af5/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:36.070153 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83efaee-f06e-4eaf-b41d-679c940a53bf admin admin] RBAC: Authorizing `identity:create_grant(project_id=c3b0319262bc468d82fa3fba94540142, user_id=01d3c959a87247cd833db555fdef9af5, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:36.072839 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83efaee-f06e-4eaf-b41d-679c940a53bf admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:36.142364 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1375/2720] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:48:36 2022] PUT /identity/v3/projects/c3b0319262bc468d82fa3fba94540142/users/01d3c959a87247cd833db555fdef9af5/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:48:36.148078 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7dc2c1-044e-4aea-b78b-00397fd45f94 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:36.148665 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7dc2c1-044e-4aea-b78b-00397fd45f94 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:36.149139 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7dc2c1-044e-4aea-b78b-00397fd45f94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:36.167712 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-ab7dc2c1-044e-4aea-b78b-00397fd45f94 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:48:36.168967 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-ab7dc2c1-044e-4aea-b78b-00397fd45f94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:36.230580 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1346/2721] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:48:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:48:36.240383 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83efaee-f06e-4eaf-b41d-679c940a53bf admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:36.242114 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-f83efaee-f06e-4eaf-b41d-679c940a53bf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:36.299353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83efaee-f06e-4eaf-b41d-679c940a53bf admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:36.303141 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0736e0-0a8d-497f-bb03-5c11f78fa7f5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:36.303782 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0736e0-0a8d-497f-bb03-5c11f78fa7f5 service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:36.304069 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0736e0-0a8d-497f-bb03-5c11f78fa7f5 service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:36.304353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0736e0-0a8d-497f-bb03-5c11f78fa7f5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:36.313799 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0736e0-0a8d-497f-bb03-5c11f78fa7f5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:36.315256 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0736e0-0a8d-497f-bb03-5c11f78fa7f5 service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:36.325412 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1376/2722] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:48:36 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:48:37.019287 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7dc2c1-044e-4aea-b78b-00397fd45f94 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:37.020909 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-ab7dc2c1-044e-4aea-b78b-00397fd45f94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:37.034676 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7dc2c1-044e-4aea-b78b-00397fd45f94 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:37.040110 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0ea11a-fad1-47e7-a348-67dfdc1867b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:37.041068 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0ea11a-fad1-47e7-a348-67dfdc1867b0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:37.042018 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0ea11a-fad1-47e7-a348-67dfdc1867b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:37.042658 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c0ea11a-fad1-47e7-a348-67dfdc1867b0 admin admin] PATH_INFO: `/v3/users/756d496daf0f4428b2157d6afdff2e80` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:37.043985 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c0ea11a-fad1-47e7-a348-67dfdc1867b0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=756d496daf0f4428b2157d6afdff2e80)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:37.046031 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c0ea11a-fad1-47e7-a348-67dfdc1867b0 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:37.243814 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-8c0ea11a-fad1-47e7-a348-67dfdc1867b0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '756d496daf0f4428b2157d6afdff2e80'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:37.315267 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-8c0ea11a-fad1-47e7-a348-67dfdc1867b0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '756d496daf0f4428b2157d6afdff2e80'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:37.319331 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-8c0ea11a-fad1-47e7-a348-67dfdc1867b0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '756d496daf0f4428b2157d6afdff2e80'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:37.324421 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-8c0ea11a-fad1-47e7-a348-67dfdc1867b0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '756d496daf0f4428b2157d6afdff2e80'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:37.337821 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-8c0ea11a-fad1-47e7-a348-67dfdc1867b0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '756d496daf0f4428b2157d6afdff2e80'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:37.344465 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1347/2723] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:48:37 2022] DELETE /identity/v3/users/756d496daf0f4428b2157d6afdff2e80 => generated 0 bytes in 326 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:48:37.349051 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa64098-2a76-4d7c-bd1b-3b118f01cd7a None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:37.349759 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa64098-2a76-4d7c-bd1b-3b118f01cd7a None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:37.349759 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa64098-2a76-4d7c-bd1b-3b118f01cd7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:37.371744 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-2fa64098-2a76-4d7c-bd1b-3b118f01cd7a None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:48:37.373034 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-2fa64098-2a76-4d7c-bd1b-3b118f01cd7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:37.455831 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1377/2724] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:48:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:48:37.465968 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0ea11a-fad1-47e7-a348-67dfdc1867b0 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:37.467445 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-8c0ea11a-fad1-47e7-a348-67dfdc1867b0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:37.478594 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:48:37.539867 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c0ea11a-fad1-47e7-a348-67dfdc1867b0 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:37.543882 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c30f44-c904-4223-bece-a609e646d4c8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:37.548492 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c30f44-c904-4223-bece-a609e646d4c8 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:37.548492 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c30f44-c904-4223-bece-a609e646d4c8 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:37.548492 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c30f44-c904-4223-bece-a609e646d4c8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:37.555184 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c30f44-c904-4223-bece-a609e646d4c8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:37.556628 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c30f44-c904-4223-bece-a609e646d4c8 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:37.572318 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1348/2725] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:48:37 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:48:37.688168 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968b0dfe-b1eb-4e9b-b5c8-65288f88b953 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:37.688512 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968b0dfe-b1eb-4e9b-b5c8-65288f88b953 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:37.688813 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968b0dfe-b1eb-4e9b-b5c8-65288f88b953 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:37.718032 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-968b0dfe-b1eb-4e9b-b5c8-65288f88b953 None None] MFA Rules not processed for user `01d3c959a87247cd833db555fdef9af5`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:48:37.719558 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-968b0dfe-b1eb-4e9b-b5c8-65288f88b953 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:37.792085 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1378/2726] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:48:37 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:48:37.803071 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c30f44-c904-4223-bece-a609e646d4c8 service neutron] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:37.804617 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-09c30f44-c904-4223-bece-a609e646d4c8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:37.897067 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c30f44-c904-4223-bece-a609e646d4c8 service neutron] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:37.901935 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d5e818-43e2-4ea8-8cb9-045b205cbe6b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:37.902849 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d5e818-43e2-4ea8-8cb9-045b205cbe6b service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:37.903284 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d5e818-43e2-4ea8-8cb9-045b205cbe6b service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:37.903424 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d5e818-43e2-4ea8-8cb9-045b205cbe6b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:37.913049 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d5e818-43e2-4ea8-8cb9-045b205cbe6b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:37.914531 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d5e818-43e2-4ea8-8cb9-045b205cbe6b service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:37.924128 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1349/2727] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:48:37 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:48:38.083054 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968b0dfe-b1eb-4e9b-b5c8-65288f88b953 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:38.084676 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-968b0dfe-b1eb-4e9b-b5c8-65288f88b953 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:38.093066 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:48:38.100010 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968b0dfe-b1eb-4e9b-b5c8-65288f88b953 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:38.104778 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a40be0c-46a7-424d-8ba6-aed40ab7181f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:38.105858 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a40be0c-46a7-424d-8ba6-aed40ab7181f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:38.106307 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a40be0c-46a7-424d-8ba6-aed40ab7181f admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:38.106537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a40be0c-46a7-424d-8ba6-aed40ab7181f admin admin] PATH_INFO: `/v3/projects/fa7e2c910ccf4fa3b94cd780a0011ef8` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:38.107747 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a40be0c-46a7-424d-8ba6-aed40ab7181f admin admin] RBAC: Authorizing `identity:delete_project(project_id=fa7e2c910ccf4fa3b94cd780a0011ef8)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:38.109827 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a40be0c-46a7-424d-8ba6-aed40ab7181f admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:38.265845 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-5a40be0c-46a7-424d-8ba6-aed40ab7181f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fa7e2c910ccf4fa3b94cd780a0011ef8', 'request_id': 'req-5a40be0c-46a7-424d-8ba6-aed40ab7181f'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:38.271838 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-5a40be0c-46a7-424d-8ba6-aed40ab7181f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fa7e2c910ccf4fa3b94cd780a0011ef8', 'request_id': 'req-5a40be0c-46a7-424d-8ba6-aed40ab7181f'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:38.378586 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-5a40be0c-46a7-424d-8ba6-aed40ab7181f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fa7e2c910ccf4fa3b94cd780a0011ef8', 'request_id': 'req-5a40be0c-46a7-424d-8ba6-aed40ab7181f'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:38.393401 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-5a40be0c-46a7-424d-8ba6-aed40ab7181f admin admin] The token cache is being invalidate because project fa7e2c910ccf4fa3b94cd780a0011ef8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:48:38.393700 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-5a40be0c-46a7-424d-8ba6-aed40ab7181f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:38.394371 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-5a40be0c-46a7-424d-8ba6-aed40ab7181f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:38.395983 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1379/2728] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:48:38 2022] DELETE /identity/v3/projects/fa7e2c910ccf4fa3b94cd780a0011ef8 => generated 0 bytes in 314 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:48:38.400195 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d5e818-43e2-4ea8-8cb9-045b205cbe6b service cinder] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:38.411514 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d5e818-43e2-4ea8-8cb9-045b205cbe6b service cinder] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:38.469848 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc749f9-5d23-4db3-b09a-28de449b0261 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:38.470762 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc749f9-5d23-4db3-b09a-28de449b0261 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:38.471046 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc749f9-5d23-4db3-b09a-28de449b0261 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:38.471447 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc749f9-5d23-4db3-b09a-28de449b0261 admin admin] PATH_INFO: `/v3/users/6747fe9912ab4e4791b81047650b9b3b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:38.472635 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc749f9-5d23-4db3-b09a-28de449b0261 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6747fe9912ab4e4791b81047650b9b3b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:38.475244 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc749f9-5d23-4db3-b09a-28de449b0261 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:38.731173 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-8dc749f9-5d23-4db3-b09a-28de449b0261 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6747fe9912ab4e4791b81047650b9b3b'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:38.847309 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-8dc749f9-5d23-4db3-b09a-28de449b0261 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6747fe9912ab4e4791b81047650b9b3b'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:38.847796 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-8dc749f9-5d23-4db3-b09a-28de449b0261 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6747fe9912ab4e4791b81047650b9b3b'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:38.850914 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:48:38.856331 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-8dc749f9-5d23-4db3-b09a-28de449b0261 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6747fe9912ab4e4791b81047650b9b3b'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:38.868718 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-8dc749f9-5d23-4db3-b09a-28de449b0261 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6747fe9912ab4e4791b81047650b9b3b'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:38.875004 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1350/2729] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:48:38 2022] DELETE /identity/v3/users/6747fe9912ab4e4791b81047650b9b3b => generated 0 bytes in 475 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:48:39.282867 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a40be0c-46a7-424d-8ba6-aed40ab7181f admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:48:39.284358 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-5a40be0c-46a7-424d-8ba6-aed40ab7181f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:39.297873 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a40be0c-46a7-424d-8ba6-aed40ab7181f admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:48:39.360213 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9afe15-df9f-4499-9a2c-3ad4b83eb034 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:48:39.361090 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9afe15-df9f-4499-9a2c-3ad4b83eb034 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:39.361987 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9afe15-df9f-4499-9a2c-3ad4b83eb034 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:39.361987 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9afe15-df9f-4499-9a2c-3ad4b83eb034 admin admin] PATH_INFO: `/v3/projects/299dcb61892d4e9a8b2c023bc157c37c` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:39.362890 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9afe15-df9f-4499-9a2c-3ad4b83eb034 admin admin] RBAC: Authorizing `identity:delete_project(project_id=299dcb61892d4e9a8b2c023bc157c37c)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:48:39.365422 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9afe15-df9f-4499-9a2c-3ad4b83eb034 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:48:39.508995 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-da9afe15-df9f-4499-9a2c-3ad4b83eb034 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '299dcb61892d4e9a8b2c023bc157c37c', 'request_id': 'req-da9afe15-df9f-4499-9a2c-3ad4b83eb034'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:39.515537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-da9afe15-df9f-4499-9a2c-3ad4b83eb034 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '299dcb61892d4e9a8b2c023bc157c37c', 'request_id': 'req-da9afe15-df9f-4499-9a2c-3ad4b83eb034'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:39.617783 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-da9afe15-df9f-4499-9a2c-3ad4b83eb034 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '299dcb61892d4e9a8b2c023bc157c37c', 'request_id': 'req-da9afe15-df9f-4499-9a2c-3ad4b83eb034'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:39.634556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-da9afe15-df9f-4499-9a2c-3ad4b83eb034 admin admin] The token cache is being invalidate because project 299dcb61892d4e9a8b2c023bc157c37c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:48:39.634857 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-da9afe15-df9f-4499-9a2c-3ad4b83eb034 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:39.635873 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-da9afe15-df9f-4499-9a2c-3ad4b83eb034 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:48:39.650495 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1380/2730] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:48:39 2022] DELETE /identity/v3/projects/299dcb61892d4e9a8b2c023bc157c37c => generated 0 bytes in 356 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:48:39.736386 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156636cb-de41-48da-b7d9-e737a1d7df60 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:48:39.736962 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156636cb-de41-48da-b7d9-e737a1d7df60 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:48:39.736962 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156636cb-de41-48da-b7d9-e737a1d7df60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:48:39.758916 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-156636cb-de41-48da-b7d9-e737a1d7df60 None None] MFA Rules not processed for user `01d3c959a87247cd833db555fdef9af5`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:48:39.760350 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-156636cb-de41-48da-b7d9-e737a1d7df60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:48:39.830740 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1351/2731] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:48:39 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:48:49.628966 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=77,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:48:49.822385 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=35,keystone:DELETE=2 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:49:11.115451 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9afe15-df9f-4499-9a2c-3ad4b83eb034 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:49:11.116831 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-da9afe15-df9f-4499-9a2c-3ad4b83eb034 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:49:11.177209 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9afe15-df9f-4499-9a2c-3ad4b83eb034 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:49:11.180777 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54129b6-7bf2-4482-a5cd-defa651e088a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:49:11.181412 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54129b6-7bf2-4482-a5cd-defa651e088a service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:49:11.181696 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54129b6-7bf2-4482-a5cd-defa651e088a service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:49:11.181973 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54129b6-7bf2-4482-a5cd-defa651e088a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:49:11.183734 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-b54129b6-7bf2-4482-a5cd-defa651e088a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:49:11.253729 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b54129b6-7bf2-4482-a5cd-defa651e088a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:49:11.255207 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b54129b6-7bf2-4482-a5cd-defa651e088a service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:49:11.266880 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1381/2732] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:49:11 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:49:21.261314 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=54 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:49:36.796912 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156636cb-de41-48da-b7d9-e737a1d7df60 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:49:36.798713 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-156636cb-de41-48da-b7d9-e737a1d7df60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:49:36.811547 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156636cb-de41-48da-b7d9-e737a1d7df60 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:49:36.816044 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-531ab3fd-36b0-48e9-9d7e-082fdda85b1b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:49:36.816966 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531ab3fd-36b0-48e9-9d7e-082fdda85b1b service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:49:36.817256 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531ab3fd-36b0-48e9-9d7e-082fdda85b1b service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:49:36.817540 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531ab3fd-36b0-48e9-9d7e-082fdda85b1b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:49:36.819356 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-531ab3fd-36b0-48e9-9d7e-082fdda85b1b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:49:36.834601 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-531ab3fd-36b0-48e9-9d7e-082fdda85b1b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:49:36.836073 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-531ab3fd-36b0-48e9-9d7e-082fdda85b1b service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:49:36.848376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1352/2733] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:49:36 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:49:46.842789 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=6 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:49:50.846838 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54129b6-7bf2-4482-a5cd-defa651e088a service cinder] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:49:50.848420 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-b54129b6-7bf2-4482-a5cd-defa651e088a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:49:50.863200 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54129b6-7bf2-4482-a5cd-defa651e088a service cinder] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:49:50.868808 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4727d0-b437-4b5d-affc-513a2eb12a2c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:49:50.869454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4727d0-b437-4b5d-affc-513a2eb12a2c service nova] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:49:50.869742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4727d0-b437-4b5d-affc-513a2eb12a2c service nova] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:49:50.870019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4727d0-b437-4b5d-affc-513a2eb12a2c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:49:50.871866 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-fe4727d0-b437-4b5d-affc-513a2eb12a2c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:49:50.885990 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe4727d0-b437-4b5d-affc-513a2eb12a2c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:49:50.887840 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe4727d0-b437-4b5d-affc-513a2eb12a2c service nova] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:49:50.899301 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1382/2734] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:49:50 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:49:54.868522 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49fdd48-559b-4f1f-a159-55f00d90b3b2 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:49:54.869118 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49fdd48-559b-4f1f-a159-55f00d90b3b2 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:49:54.869118 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49fdd48-559b-4f1f-a159-55f00d90b3b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:49:54.887435 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-e49fdd48-559b-4f1f-a159-55f00d90b3b2 None None] MFA Rules not processed for user `01d3c959a87247cd833db555fdef9af5`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:49:54.888661 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-e49fdd48-559b-4f1f-a159-55f00d90b3b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:49:54.896526 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1353/2735] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:49:54 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:49:55.535252 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4727d0-b437-4b5d-affc-513a2eb12a2c service nova] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:49:55.536853 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-fe4727d0-b437-4b5d-affc-513a2eb12a2c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:49:55.638507 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4727d0-b437-4b5d-affc-513a2eb12a2c service nova] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:49:55.644548 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90341fb1-e835-465c-a773-2ab975bd774f service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f623be28c83247c78d3ea6873c347fe0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:49:55.645207 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90341fb1-e835-465c-a773-2ab975bd774f service placement] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:49:55.645497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90341fb1-e835-465c-a773-2ab975bd774f service placement] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:49:55.645937 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90341fb1-e835-465c-a773-2ab975bd774f service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:49:55.647647 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-90341fb1-e835-465c-a773-2ab975bd774f service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:49:55.661226 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90341fb1-e835-465c-a773-2ab975bd774f service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:49:55.662929 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90341fb1-e835-465c-a773-2ab975bd774f service placement] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:49:55.678478 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1383/2736] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:49:55 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:49:57.771079 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49fdd48-559b-4f1f-a159-55f00d90b3b2 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:49:57.773461 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-e49fdd48-559b-4f1f-a159-55f00d90b3b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:49:57.848910 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49fdd48-559b-4f1f-a159-55f00d90b3b2 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:49:57.852648 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d288c4-9171-42da-95e5-20c7400da11e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:49:57.853323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d288c4-9171-42da-95e5-20c7400da11e service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:49:57.853757 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d288c4-9171-42da-95e5-20c7400da11e service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:49:57.854044 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d288c4-9171-42da-95e5-20c7400da11e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:49:57.856150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-e0d288c4-9171-42da-95e5-20c7400da11e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:49:57.926574 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d288c4-9171-42da-95e5-20c7400da11e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:49:57.928036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d288c4-9171-42da-95e5-20c7400da11e service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:49:57.940242 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1354/2737] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:49:57 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 171 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:50:00.229508 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90341fb1-e835-465c-a773-2ab975bd774f service placement] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:00.231839 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-90341fb1-e835-465c-a773-2ab975bd774f service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:00.244309 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90341fb1-e835-465c-a773-2ab975bd774f service placement] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:00.248406 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead2c962-0e65-4705-9b28-7f86c65be07f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:00.249133 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead2c962-0e65-4705-9b28-7f86c65be07f service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:00.249658 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead2c962-0e65-4705-9b28-7f86c65be07f service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:00.251372 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead2c962-0e65-4705-9b28-7f86c65be07f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:00.254591 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-ead2c962-0e65-4705-9b28-7f86c65be07f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:00.269876 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead2c962-0e65-4705-9b28-7f86c65be07f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:00.271553 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead2c962-0e65-4705-9b28-7f86c65be07f service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:00.281532 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1384/2738] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:50:00 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:50:02.424755 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d288c4-9171-42da-95e5-20c7400da11e service neutron] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:02.426398 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-e0d288c4-9171-42da-95e5-20c7400da11e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:02.443954 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d288c4-9171-42da-95e5-20c7400da11e service neutron] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:02.451904 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94394a28-5094-4ce4-a4f2-b81d40810264 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:02.453920 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94394a28-5094-4ce4-a4f2-b81d40810264 service nova] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:02.453920 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94394a28-5094-4ce4-a4f2-b81d40810264 service nova] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:02.453920 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94394a28-5094-4ce4-a4f2-b81d40810264 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:02.455809 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-94394a28-5094-4ce4-a4f2-b81d40810264 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:02.476455 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94394a28-5094-4ce4-a4f2-b81d40810264 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:02.478848 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94394a28-5094-4ce4-a4f2-b81d40810264 service nova] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:02.511399 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1355/2739] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:50:02 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:50:06.375313 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead2c962-0e65-4705-9b28-7f86c65be07f service neutron] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:06.380585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-ead2c962-0e65-4705-9b28-7f86c65be07f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:06.398748 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead2c962-0e65-4705-9b28-7f86c65be07f service neutron] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:06.403460 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ee5c50-4613-4c54-b1d8-79bc12604207 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:06.404525 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ee5c50-4613-4c54-b1d8-79bc12604207 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:06.404837 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ee5c50-4613-4c54-b1d8-79bc12604207 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:06.409928 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ee5c50-4613-4c54-b1d8-79bc12604207 admin admin] PATH_INFO: `/v3/users/bde6ee9df10146d98991b0d4bb37fc48` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:06.409928 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ee5c50-4613-4c54-b1d8-79bc12604207 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bde6ee9df10146d98991b0d4bb37fc48)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:06.409928 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ee5c50-4613-4c54-b1d8-79bc12604207 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:06.609161 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-a4ee5c50-4613-4c54-b1d8-79bc12604207 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bde6ee9df10146d98991b0d4bb37fc48'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:06.609812 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-a4ee5c50-4613-4c54-b1d8-79bc12604207 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bde6ee9df10146d98991b0d4bb37fc48'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:06.610528 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-a4ee5c50-4613-4c54-b1d8-79bc12604207 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bde6ee9df10146d98991b0d4bb37fc48'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:06.617643 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-a4ee5c50-4613-4c54-b1d8-79bc12604207 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bde6ee9df10146d98991b0d4bb37fc48'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:06.630403 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-a4ee5c50-4613-4c54-b1d8-79bc12604207 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bde6ee9df10146d98991b0d4bb37fc48'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:06.707212 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1385/2740] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:50:06 2022] DELETE /identity/v3/users/bde6ee9df10146d98991b0d4bb37fc48 => generated 0 bytes in 332 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:50:06.981376 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94394a28-5094-4ce4-a4f2-b81d40810264 service nova] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:06.993071 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94394a28-5094-4ce4-a4f2-b81d40810264 service nova] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:07.055940 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831ba36e-db24-430c-8765-48d483f657e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:07.056954 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831ba36e-db24-430c-8765-48d483f657e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:07.057243 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831ba36e-db24-430c-8765-48d483f657e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:07.057534 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831ba36e-db24-430c-8765-48d483f657e7 admin admin] PATH_INFO: `/v3/projects/e4b2b488b746454b9f93d67fa3c97809` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:07.058852 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-831ba36e-db24-430c-8765-48d483f657e7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e4b2b488b746454b9f93d67fa3c97809)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:07.060743 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-831ba36e-db24-430c-8765-48d483f657e7 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:07.172248 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:50:07.185307 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-831ba36e-db24-430c-8765-48d483f657e7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e4b2b488b746454b9f93d67fa3c97809', 'request_id': 'req-831ba36e-db24-430c-8765-48d483f657e7'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:07.192803 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-831ba36e-db24-430c-8765-48d483f657e7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e4b2b488b746454b9f93d67fa3c97809', 'request_id': 'req-831ba36e-db24-430c-8765-48d483f657e7'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:07.199127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-831ba36e-db24-430c-8765-48d483f657e7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e4b2b488b746454b9f93d67fa3c97809', 'request_id': 'req-831ba36e-db24-430c-8765-48d483f657e7'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:07.232903 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ee5c50-4613-4c54-b1d8-79bc12604207 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:07.234314 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-a4ee5c50-4613-4c54-b1d8-79bc12604207 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:07.272790 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-831ba36e-db24-430c-8765-48d483f657e7 admin admin] The token cache is being invalidate because project e4b2b488b746454b9f93d67fa3c97809 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:50:07.273093 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-831ba36e-db24-430c-8765-48d483f657e7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:07.274088 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-831ba36e-db24-430c-8765-48d483f657e7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:07.275690 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1356/2741] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:50:06 2022] DELETE /identity/v3/projects/e4b2b488b746454b9f93d67fa3c97809 => generated 0 bytes in 295 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:50:07.276758 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:50:07.279719 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831ba36e-db24-430c-8765-48d483f657e7 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:07.281102 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-831ba36e-db24-430c-8765-48d483f657e7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:07.319484 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ee5c50-4613-4c54-b1d8-79bc12604207 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:07.366278 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831ba36e-db24-430c-8765-48d483f657e7 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:07.374348 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e3aeb4-1f99-4b15-b316-e5c1fd9c4a04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:07.374348 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e3aeb4-1f99-4b15-b316-e5c1fd9c4a04 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:07.374348 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e3aeb4-1f99-4b15-b316-e5c1fd9c4a04 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:07.374348 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1e3aeb4-1f99-4b15-b316-e5c1fd9c4a04 admin admin] PATH_INFO: `/v3/users/c23af560b5384d6cbae44ed59ee28bbb` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:07.374809 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e3aeb4-1f99-4b15-b316-e5c1fd9c4a04 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c23af560b5384d6cbae44ed59ee28bbb)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:07.376288 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1e3aeb4-1f99-4b15-b316-e5c1fd9c4a04 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:07.404479 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d005c5-e9b4-4d15-91bb-1cb952dfa080 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:07.405136 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d005c5-e9b4-4d15-91bb-1cb952dfa080 service nova] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:07.405428 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d005c5-e9b4-4d15-91bb-1cb952dfa080 service nova] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:07.405712 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d005c5-e9b4-4d15-91bb-1cb952dfa080 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:07.407556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-08d005c5-e9b4-4d15-91bb-1cb952dfa080 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:07.420068 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08d005c5-e9b4-4d15-91bb-1cb952dfa080 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:07.421700 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08d005c5-e9b4-4d15-91bb-1cb952dfa080 service nova] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:07.433226 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1386/2742] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:50:07 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 201 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 12 17:50:07.641029 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-e1e3aeb4-1f99-4b15-b316-e5c1fd9c4a04 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c23af560b5384d6cbae44ed59ee28bbb'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:07.757633 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-e1e3aeb4-1f99-4b15-b316-e5c1fd9c4a04 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c23af560b5384d6cbae44ed59ee28bbb'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:07.758197 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-e1e3aeb4-1f99-4b15-b316-e5c1fd9c4a04 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c23af560b5384d6cbae44ed59ee28bbb'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:07.765951 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-e1e3aeb4-1f99-4b15-b316-e5c1fd9c4a04 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c23af560b5384d6cbae44ed59ee28bbb'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:07.778265 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-e1e3aeb4-1f99-4b15-b316-e5c1fd9c4a04 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c23af560b5384d6cbae44ed59ee28bbb'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:07.784450 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1357/2743] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:50:07 2022] DELETE /identity/v3/users/c23af560b5384d6cbae44ed59ee28bbb => generated 0 bytes in 505 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:50:08.120300 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d005c5-e9b4-4d15-91bb-1cb952dfa080 service nova] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:08.133627 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d005c5-e9b4-4d15-91bb-1cb952dfa080 service nova] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:08.199527 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4c703a-8549-40f5-92aa-fb3b02a4b97f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:08.200391 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4c703a-8549-40f5-92aa-fb3b02a4b97f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:08.200681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4c703a-8549-40f5-92aa-fb3b02a4b97f admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:08.200962 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4c703a-8549-40f5-92aa-fb3b02a4b97f admin admin] PATH_INFO: `/v3/projects/a328dd35ab014be7acdb9605e42b7be4` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:08.202210 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4c703a-8549-40f5-92aa-fb3b02a4b97f admin admin] RBAC: Authorizing `identity:delete_project(project_id=a328dd35ab014be7acdb9605e42b7be4)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:08.205010 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4c703a-8549-40f5-92aa-fb3b02a4b97f admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:08.316578 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-1e4c703a-8549-40f5-92aa-fb3b02a4b97f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a328dd35ab014be7acdb9605e42b7be4', 'request_id': 'req-1e4c703a-8549-40f5-92aa-fb3b02a4b97f'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:08.322145 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-1e4c703a-8549-40f5-92aa-fb3b02a4b97f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a328dd35ab014be7acdb9605e42b7be4', 'request_id': 'req-1e4c703a-8549-40f5-92aa-fb3b02a4b97f'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:08.324668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:50:08.397776 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-1e4c703a-8549-40f5-92aa-fb3b02a4b97f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a328dd35ab014be7acdb9605e42b7be4', 'request_id': 'req-1e4c703a-8549-40f5-92aa-fb3b02a4b97f'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:08.409640 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-1e4c703a-8549-40f5-92aa-fb3b02a4b97f admin admin] The token cache is being invalidate because project a328dd35ab014be7acdb9605e42b7be4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:50:08.410817 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-1e4c703a-8549-40f5-92aa-fb3b02a4b97f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:08.411119 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-1e4c703a-8549-40f5-92aa-fb3b02a4b97f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:08.412692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1387/2744] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:50:08 2022] DELETE /identity/v3/projects/a328dd35ab014be7acdb9605e42b7be4 => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:50:10.030681 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e3aeb4-1f99-4b15-b316-e5c1fd9c4a04 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:10.032966 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-e1e3aeb4-1f99-4b15-b316-e5c1fd9c4a04 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:10.081526 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:50:10.096911 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1e3aeb4-1f99-4b15-b316-e5c1fd9c4a04 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:10.147817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332008ef-d39b-474b-9802-a188fad808c9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:10.148478 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332008ef-d39b-474b-9802-a188fad808c9 service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:10.148769 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332008ef-d39b-474b-9802-a188fad808c9 service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:10.149046 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332008ef-d39b-474b-9802-a188fad808c9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:10.150995 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-332008ef-d39b-474b-9802-a188fad808c9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:10.216951 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-332008ef-d39b-474b-9802-a188fad808c9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:10.218423 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-332008ef-d39b-474b-9802-a188fad808c9 service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:10.229688 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1358/2745] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:50:10 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 200 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:50:10.329631 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4c703a-8549-40f5-92aa-fb3b02a4b97f admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:10.331127 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-1e4c703a-8549-40f5-92aa-fb3b02a4b97f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:10.391519 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4c703a-8549-40f5-92aa-fb3b02a4b97f admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:10.395363 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542415ad-0d33-4bf5-9e47-8b02c06e8a07 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:10.396027 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542415ad-0d33-4bf5-9e47-8b02c06e8a07 service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:10.396320 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542415ad-0d33-4bf5-9e47-8b02c06e8a07 service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:10.396602 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-542415ad-0d33-4bf5-9e47-8b02c06e8a07 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:10.398467 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-542415ad-0d33-4bf5-9e47-8b02c06e8a07 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:10.461823 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-542415ad-0d33-4bf5-9e47-8b02c06e8a07 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:10.463301 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-542415ad-0d33-4bf5-9e47-8b02c06e8a07 service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:10.474188 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1388/2746] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:50:10 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:50:14.392855 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332008ef-d39b-474b-9802-a188fad808c9 service cinder] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:14.395332 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-332008ef-d39b-474b-9802-a188fad808c9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:14.469753 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332008ef-d39b-474b-9802-a188fad808c9 service cinder] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:14.475731 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14bec582-964d-405e-b09c-d2ffd792576e service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f623be28c83247c78d3ea6873c347fe0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:14.476415 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14bec582-964d-405e-b09c-d2ffd792576e service placement] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:14.476736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14bec582-964d-405e-b09c-d2ffd792576e service placement] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:14.477043 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14bec582-964d-405e-b09c-d2ffd792576e service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:14.478850 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-14bec582-964d-405e-b09c-d2ffd792576e service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:14.491049 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14bec582-964d-405e-b09c-d2ffd792576e service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:14.492660 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14bec582-964d-405e-b09c-d2ffd792576e service placement] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:14.502781 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1359/2747] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:50:14 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:50:14.546772 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542415ad-0d33-4bf5-9e47-8b02c06e8a07 service neutron] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:14.548221 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-542415ad-0d33-4bf5-9e47-8b02c06e8a07 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:14.561531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-542415ad-0d33-4bf5-9e47-8b02c06e8a07 service neutron] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:14.566821 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58d8937-6d04-4c52-97a8-a2b68289f501 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:14.567742 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58d8937-6d04-4c52-97a8-a2b68289f501 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:14.568032 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58d8937-6d04-4c52-97a8-a2b68289f501 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:14.568315 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58d8937-6d04-4c52-97a8-a2b68289f501 admin admin] PATH_INFO: `/v3/users/e3d86a1cde9f407cae8ac050fe000d47` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:14.569577 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58d8937-6d04-4c52-97a8-a2b68289f501 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e3d86a1cde9f407cae8ac050fe000d47)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:14.571694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58d8937-6d04-4c52-97a8-a2b68289f501 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:14.769288 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-f58d8937-6d04-4c52-97a8-a2b68289f501 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e3d86a1cde9f407cae8ac050fe000d47'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:14.770073 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-f58d8937-6d04-4c52-97a8-a2b68289f501 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e3d86a1cde9f407cae8ac050fe000d47'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:14.770670 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-f58d8937-6d04-4c52-97a8-a2b68289f501 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e3d86a1cde9f407cae8ac050fe000d47'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:14.777955 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-f58d8937-6d04-4c52-97a8-a2b68289f501 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e3d86a1cde9f407cae8ac050fe000d47'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:14.789852 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-f58d8937-6d04-4c52-97a8-a2b68289f501 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e3d86a1cde9f407cae8ac050fe000d47'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:14.872009 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1389/2748] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:50:14 2022] DELETE /identity/v3/users/e3d86a1cde9f407cae8ac050fe000d47 => generated 0 bytes in 326 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:50:15.132778 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14bec582-964d-405e-b09c-d2ffd792576e service placement] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:15.143634 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14bec582-964d-405e-b09c-d2ffd792576e service placement] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:15.186183 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:50:15.206053 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1326d7d4-0d37-4c93-a39f-3537e48dc181 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:15.207281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1326d7d4-0d37-4c93-a39f-3537e48dc181 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:15.207600 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1326d7d4-0d37-4c93-a39f-3537e48dc181 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:15.207885 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1326d7d4-0d37-4c93-a39f-3537e48dc181 admin admin] PATH_INFO: `/v3/projects/041c2c26c82a44d98d2229969c1f3618` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:15.209173 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1326d7d4-0d37-4c93-a39f-3537e48dc181 admin admin] RBAC: Authorizing `identity:delete_project(project_id=041c2c26c82a44d98d2229969c1f3618)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:15.211236 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1326d7d4-0d37-4c93-a39f-3537e48dc181 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:15.323560 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-1326d7d4-0d37-4c93-a39f-3537e48dc181 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '041c2c26c82a44d98d2229969c1f3618', 'request_id': 'req-1326d7d4-0d37-4c93-a39f-3537e48dc181'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:15.329945 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-1326d7d4-0d37-4c93-a39f-3537e48dc181 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '041c2c26c82a44d98d2229969c1f3618', 'request_id': 'req-1326d7d4-0d37-4c93-a39f-3537e48dc181'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:15.336498 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-1326d7d4-0d37-4c93-a39f-3537e48dc181 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '041c2c26c82a44d98d2229969c1f3618', 'request_id': 'req-1326d7d4-0d37-4c93-a39f-3537e48dc181'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:15.449365 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-1326d7d4-0d37-4c93-a39f-3537e48dc181 admin admin] The token cache is being invalidate because project 041c2c26c82a44d98d2229969c1f3618 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:50:15.450046 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-1326d7d4-0d37-4c93-a39f-3537e48dc181 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:15.450393 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-1326d7d4-0d37-4c93-a39f-3537e48dc181 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:15.451987 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1360/2749] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:50:15 2022] DELETE /identity/v3/projects/041c2c26c82a44d98d2229969c1f3618 => generated 0 bytes in 320 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:50:15.457006 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c3b775-c3a5-4b90-94fc-53c5300bade1 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:15.457292 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c3b775-c3a5-4b90-94fc-53c5300bade1 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:15.457592 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c3b775-c3a5-4b90-94fc-53c5300bade1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:15.477096 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-a6c3b775-c3a5-4b90-94fc-53c5300bade1 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:50:15.478502 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-a6c3b775-c3a5-4b90-94fc-53c5300bade1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:15.483880 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:DELETE=11,keystone:SELECT=88,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:50:15.539299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1390/2750] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:50:15 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:50:15.547943 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a37b57-9a0c-4b50-895f-1204014da9c8 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:15.548236 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a37b57-9a0c-4b50-895f-1204014da9c8 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:15.548521 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a37b57-9a0c-4b50-895f-1204014da9c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:15.567608 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-55a37b57-9a0c-4b50-895f-1204014da9c8 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:50:15.569478 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-55a37b57-9a0c-4b50-895f-1204014da9c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:15.577114 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1361/2751] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:50:15 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:50:15.581422 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c3b775-c3a5-4b90-94fc-53c5300bade1 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:15.592615 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c3b775-c3a5-4b90-94fc-53c5300bade1 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:15.599406 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6375cf5b-a4b7-480c-8677-ba51d5496685 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:15.599645 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6375cf5b-a4b7-480c-8677-ba51d5496685 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:15.599919 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6375cf5b-a4b7-480c-8677-ba51d5496685 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:15.600213 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6375cf5b-a4b7-480c-8677-ba51d5496685 admin admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:15.600985 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6375cf5b-a4b7-480c-8677-ba51d5496685 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:50:15.601298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6375cf5b-a4b7-480c-8677-ba51d5496685 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:15.602935 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6375cf5b-a4b7-480c-8677-ba51d5496685 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:15.615952 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1391/2752] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:50:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:50:15.620159 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a37b57-9a0c-4b50-895f-1204014da9c8 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:15.629949 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a37b57-9a0c-4b50-895f-1204014da9c8 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:15.635462 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5674c1ef-db77-4136-acbb-9c9d9304043c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:15.636122 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5674c1ef-db77-4136-acbb-9c9d9304043c admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:15.636408 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5674c1ef-db77-4136-acbb-9c9d9304043c admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:15.636731 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5674c1ef-db77-4136-acbb-9c9d9304043c admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:15.637425 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5674c1ef-db77-4136-acbb-9c9d9304043c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:15.639368 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5674c1ef-db77-4136-acbb-9c9d9304043c admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:15.725123 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1362/2753] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:50:15 2022] POST /identity/v3/projects => generated 355 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:50:15.730922 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6375cf5b-a4b7-480c-8677-ba51d5496685 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:15.740047 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6375cf5b-a4b7-480c-8677-ba51d5496685 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:15.796588 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa663ce4-750a-492c-a5cc-c752f518236f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:15.797259 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa663ce4-750a-492c-a5cc-c752f518236f admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:15.797545 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa663ce4-750a-492c-a5cc-c752f518236f admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:15.797825 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa663ce4-750a-492c-a5cc-c752f518236f admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:15.798556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa663ce4-750a-492c-a5cc-c752f518236f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:15.800483 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa663ce4-750a-492c-a5cc-c752f518236f admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:15.881142 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1392/2754] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:50:15 2022] POST /identity/v3/users => generated 278 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:50:15.885163 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5674c1ef-db77-4136-acbb-9c9d9304043c admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:15.895568 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5674c1ef-db77-4136-acbb-9c9d9304043c admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:15.900340 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e54b33a-04e0-4f62-abc7-bc8a7b11f7cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:15.901026 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e54b33a-04e0-4f62-abc7-bc8a7b11f7cf admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:15.901305 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e54b33a-04e0-4f62-abc7-bc8a7b11f7cf admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:15.901581 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e54b33a-04e0-4f62-abc7-bc8a7b11f7cf admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:15.902339 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e54b33a-04e0-4f62-abc7-bc8a7b11f7cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:15.903959 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e54b33a-04e0-4f62-abc7-bc8a7b11f7cf admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:15.914870 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1363/2755] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:50:15 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:50:15.918860 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa663ce4-750a-492c-a5cc-c752f518236f admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:15.930303 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa663ce4-750a-492c-a5cc-c752f518236f admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:15.934903 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e128f76e-f84d-4e1c-9ca1-73a904b75971 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:15.935573 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e128f76e-f84d-4e1c-9ca1-73a904b75971 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:15.935845 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e128f76e-f84d-4e1c-9ca1-73a904b75971 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:15.936124 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e128f76e-f84d-4e1c-9ca1-73a904b75971 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:15.936859 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e128f76e-f84d-4e1c-9ca1-73a904b75971 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:15.938452 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e128f76e-f84d-4e1c-9ca1-73a904b75971 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:15.948818 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1393/2756] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:50:15 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:50:15.952988 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e54b33a-04e0-4f62-abc7-bc8a7b11f7cf admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:15.966749 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e54b33a-04e0-4f62-abc7-bc8a7b11f7cf admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:15.974329 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e98d98-9edb-4de2-a2bd-f423f12d5dfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:15.975346 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e98d98-9edb-4de2-a2bd-f423f12d5dfb admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:15.975578 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e98d98-9edb-4de2-a2bd-f423f12d5dfb admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:15.975859 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e98d98-9edb-4de2-a2bd-f423f12d5dfb admin admin] PATH_INFO: `/v3/projects/cf732a5a623d4cc6a7e3cddb43c417d6/users/0d6e31538a92446da2beb8e7c4298330/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:15.991781 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e98d98-9edb-4de2-a2bd-f423f12d5dfb admin admin] RBAC: Authorizing `identity:create_grant(project_id=cf732a5a623d4cc6a7e3cddb43c417d6, user_id=0d6e31538a92446da2beb8e7c4298330, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:15.994512 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e98d98-9edb-4de2-a2bd-f423f12d5dfb admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:16.061127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1364/2757] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:50:15 2022] PUT /identity/v3/projects/cf732a5a623d4cc6a7e3cddb43c417d6/users/0d6e31538a92446da2beb8e7c4298330/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:50:16.071337 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1677650c-0330-4234-a9ed-f843076bddda None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:16.071653 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1677650c-0330-4234-a9ed-f843076bddda None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:16.072023 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1677650c-0330-4234-a9ed-f843076bddda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:16.099831 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-1677650c-0330-4234-a9ed-f843076bddda None None] MFA Rules not processed for user `0d6e31538a92446da2beb8e7c4298330`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:50:16.101051 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-1677650c-0330-4234-a9ed-f843076bddda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:16.164785 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1394/2758] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:50:16 2022] POST /identity/v3/auth/tokens => generated 3222 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:50:16.169209 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e98d98-9edb-4de2-a2bd-f423f12d5dfb admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:16.229116 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:50:16.232753 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e98d98-9edb-4de2-a2bd-f423f12d5dfb admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:16.237441 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-016dce0d-cff5-4369-8493-132895ccaca0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:16.241447 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016dce0d-cff5-4369-8493-132895ccaca0 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:16.241447 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016dce0d-cff5-4369-8493-132895ccaca0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:16.241447 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016dce0d-cff5-4369-8493-132895ccaca0 admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:16.241447 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-016dce0d-cff5-4369-8493-132895ccaca0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:16.241904 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-016dce0d-cff5-4369-8493-132895ccaca0 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:16.325357 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1365/2759] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:50:16 2022] POST /identity/v3/projects => generated 357 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:50:16.329506 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1677650c-0330-4234-a9ed-f843076bddda None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:16.340518 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1677650c-0330-4234-a9ed-f843076bddda None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:16.344221 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:50:16.398681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83aa9da3-e8ad-41e6-b826-4a9a4621b1ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:16.399363 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83aa9da3-e8ad-41e6-b826-4a9a4621b1ae admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:16.399653 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83aa9da3-e8ad-41e6-b826-4a9a4621b1ae admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:16.399942 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83aa9da3-e8ad-41e6-b826-4a9a4621b1ae admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:16.400667 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83aa9da3-e8ad-41e6-b826-4a9a4621b1ae admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:16.402624 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83aa9da3-e8ad-41e6-b826-4a9a4621b1ae admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:16.531389 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1395/2760] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:50:16 2022] POST /identity/v3/users => generated 285 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:50:16.535724 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-016dce0d-cff5-4369-8493-132895ccaca0 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:16.547121 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-016dce0d-cff5-4369-8493-132895ccaca0 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:16.552469 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38313d5-0a05-47f1-aafd-8814bf79eae8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:16.553205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38313d5-0a05-47f1-aafd-8814bf79eae8 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:16.553523 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38313d5-0a05-47f1-aafd-8814bf79eae8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:16.553807 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c38313d5-0a05-47f1-aafd-8814bf79eae8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:16.554600 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c38313d5-0a05-47f1-aafd-8814bf79eae8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:16.556305 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c38313d5-0a05-47f1-aafd-8814bf79eae8 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:16.568022 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1366/2761] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:50:16 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:50:16.574470 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83aa9da3-e8ad-41e6-b826-4a9a4621b1ae admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:16.586620 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83aa9da3-e8ad-41e6-b826-4a9a4621b1ae admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:16.591321 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3e97b6-9d69-43f2-8641-4303670bc47b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:16.592202 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3e97b6-9d69-43f2-8641-4303670bc47b admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:16.593102 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3e97b6-9d69-43f2-8641-4303670bc47b admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:16.593102 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3e97b6-9d69-43f2-8641-4303670bc47b admin admin] PATH_INFO: `/v3/projects/a080e8a31f8c47b384a110dee22eb979/users/c812ad0cf5e247b68534446a34f0ec0f/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:16.657455 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3e97b6-9d69-43f2-8641-4303670bc47b admin admin] RBAC: Authorizing `identity:create_grant(project_id=a080e8a31f8c47b384a110dee22eb979, user_id=c812ad0cf5e247b68534446a34f0ec0f, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:16.660286 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3e97b6-9d69-43f2-8641-4303670bc47b admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:16.743287 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1396/2762] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:50:16 2022] PUT /identity/v3/projects/a080e8a31f8c47b384a110dee22eb979/users/c812ad0cf5e247b68534446a34f0ec0f/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:50:16.747358 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38313d5-0a05-47f1-aafd-8814bf79eae8 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:16.757737 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c38313d5-0a05-47f1-aafd-8814bf79eae8 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:16.816331 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23e7247f-9683-4e57-b1fd-79e80f985552 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:16.816998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e7247f-9683-4e57-b1fd-79e80f985552 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:16.817384 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e7247f-9683-4e57-b1fd-79e80f985552 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:16.817688 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23e7247f-9683-4e57-b1fd-79e80f985552 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:16.818395 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e7247f-9683-4e57-b1fd-79e80f985552 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:16.819993 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23e7247f-9683-4e57-b1fd-79e80f985552 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:16.829598 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1367/2763] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:50:16 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:50:16.834688 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3e97b6-9d69-43f2-8641-4303670bc47b admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:16.845156 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3e97b6-9d69-43f2-8641-4303670bc47b admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:16.850592 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d21354d-319e-40cb-a244-4f4aa8cec4dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:16.851619 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d21354d-319e-40cb-a244-4f4aa8cec4dc admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:16.852124 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d21354d-319e-40cb-a244-4f4aa8cec4dc admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:16.852321 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d21354d-319e-40cb-a244-4f4aa8cec4dc admin admin] PATH_INFO: `/v3/projects/a080e8a31f8c47b384a110dee22eb979/users/c812ad0cf5e247b68534446a34f0ec0f/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:16.854490 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d21354d-319e-40cb-a244-4f4aa8cec4dc admin admin] RBAC: Authorizing `identity:create_grant(project_id=a080e8a31f8c47b384a110dee22eb979, user_id=c812ad0cf5e247b68534446a34f0ec0f, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:16.857216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d21354d-319e-40cb-a244-4f4aa8cec4dc admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:16.868129 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1397/2764] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:50:16 2022] PUT /identity/v3/projects/a080e8a31f8c47b384a110dee22eb979/users/c812ad0cf5e247b68534446a34f0ec0f/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:50:16.878455 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b4e994-03cb-4def-8e92-80f189aa1325 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:16.878757 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b4e994-03cb-4def-8e92-80f189aa1325 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:16.879043 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b4e994-03cb-4def-8e92-80f189aa1325 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:16.906412 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-33b4e994-03cb-4def-8e92-80f189aa1325 None None] MFA Rules not processed for user `c812ad0cf5e247b68534446a34f0ec0f`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:50:16.907665 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-33b4e994-03cb-4def-8e92-80f189aa1325 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:16.979440 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1368/2765] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:50:16 2022] POST /identity/v3/auth/tokens => generated 3291 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:50:16.993566 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d21354d-319e-40cb-a244-4f4aa8cec4dc admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:16.994690 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-5d21354d-319e-40cb-a244-4f4aa8cec4dc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:17.060467 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d21354d-319e-40cb-a244-4f4aa8cec4dc admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:17.064310 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f736ba-dc7b-4deb-892a-127cab94ec5f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:17.065028 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f736ba-dc7b-4deb-892a-127cab94ec5f service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:17.065247 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f736ba-dc7b-4deb-892a-127cab94ec5f service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:17.065871 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f736ba-dc7b-4deb-892a-127cab94ec5f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:17.075573 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f736ba-dc7b-4deb-892a-127cab94ec5f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:17.077010 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f736ba-dc7b-4deb-892a-127cab94ec5f service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:17.089235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1398/2766] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:50:16 2022] GET /identity/v3/auth/tokens => generated 3291 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:50:17.299611 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b4e994-03cb-4def-8e92-80f189aa1325 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:17.310257 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b4e994-03cb-4def-8e92-80f189aa1325 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:17.313981 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be90ba95-3388-4717-8278-ce23fe92d63d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:17.317363 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be90ba95-3388-4717-8278-ce23fe92d63d service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:17.317363 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be90ba95-3388-4717-8278-ce23fe92d63d service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:17.317363 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be90ba95-3388-4717-8278-ce23fe92d63d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:17.366652 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:50:17.375842 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be90ba95-3388-4717-8278-ce23fe92d63d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:17.377351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be90ba95-3388-4717-8278-ce23fe92d63d service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:17.386773 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1369/2767] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:50:17 2022] GET /identity/v3/auth/tokens => generated 3222 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:50:27.082870 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=67,keystone:INSERT=5 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:50:28.142570 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=4 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:50:41.183509 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f736ba-dc7b-4deb-892a-127cab94ec5f service cinder] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:41.248240 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f736ba-dc7b-4deb-892a-127cab94ec5f service cinder] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:41.253538 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db73550a-77f3-42a9-9bcf-57a7105b64c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:41.254446 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db73550a-77f3-42a9-9bcf-57a7105b64c8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:41.254754 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db73550a-77f3-42a9-9bcf-57a7105b64c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:41.255035 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db73550a-77f3-42a9-9bcf-57a7105b64c8 admin admin] PATH_INFO: `/v3/users/0d6e31538a92446da2beb8e7c4298330` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:41.256455 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db73550a-77f3-42a9-9bcf-57a7105b64c8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0d6e31538a92446da2beb8e7c4298330)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:41.258498 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db73550a-77f3-42a9-9bcf-57a7105b64c8 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:41.462451 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-db73550a-77f3-42a9-9bcf-57a7105b64c8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0d6e31538a92446da2beb8e7c4298330'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:41.463141 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-db73550a-77f3-42a9-9bcf-57a7105b64c8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0d6e31538a92446da2beb8e7c4298330'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:41.463753 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-db73550a-77f3-42a9-9bcf-57a7105b64c8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0d6e31538a92446da2beb8e7c4298330'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:41.470861 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-db73550a-77f3-42a9-9bcf-57a7105b64c8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0d6e31538a92446da2beb8e7c4298330'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:41.483477 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-db73550a-77f3-42a9-9bcf-57a7105b64c8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0d6e31538a92446da2beb8e7c4298330'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:41.539879 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1399/2768] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:50:41 2022] DELETE /identity/v3/users/0d6e31538a92446da2beb8e7c4298330 => generated 0 bytes in 357 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:50:41.544459 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790eee33-31fb-4a34-9304-fce0877ffb05 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:41.544922 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790eee33-31fb-4a34-9304-fce0877ffb05 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:41.545023 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790eee33-31fb-4a34-9304-fce0877ffb05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:41.562045 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-790eee33-31fb-4a34-9304-fce0877ffb05 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:50:41.563296 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-790eee33-31fb-4a34-9304-fce0877ffb05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:41.625314 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1370/2769] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:50:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:50:41.635141 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db73550a-77f3-42a9-9bcf-57a7105b64c8 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:41.636628 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-db73550a-77f3-42a9-9bcf-57a7105b64c8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:41.695813 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db73550a-77f3-42a9-9bcf-57a7105b64c8 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:41.699512 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d942fb-de94-493a-8c25-2e2651fb477b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:41.700159 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d942fb-de94-493a-8c25-2e2651fb477b service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:41.700491 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d942fb-de94-493a-8c25-2e2651fb477b service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:41.700778 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d942fb-de94-493a-8c25-2e2651fb477b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:41.710699 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d942fb-de94-493a-8c25-2e2651fb477b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:41.712132 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d942fb-de94-493a-8c25-2e2651fb477b service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:41.721615 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1400/2770] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:50:41 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:50:42.148781 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-790eee33-31fb-4a34-9304-fce0877ffb05 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:42.149569 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-790eee33-31fb-4a34-9304-fce0877ffb05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:42.165165 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-790eee33-31fb-4a34-9304-fce0877ffb05 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:42.170030 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ba6531-0245-4aa9-8ee0-793db5f58f97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:42.171788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ba6531-0245-4aa9-8ee0-793db5f58f97 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:42.172098 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ba6531-0245-4aa9-8ee0-793db5f58f97 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:42.177542 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ba6531-0245-4aa9-8ee0-793db5f58f97 admin admin] PATH_INFO: `/v3/projects/cf732a5a623d4cc6a7e3cddb43c417d6` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:42.177542 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ba6531-0245-4aa9-8ee0-793db5f58f97 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cf732a5a623d4cc6a7e3cddb43c417d6)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:42.177542 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ba6531-0245-4aa9-8ee0-793db5f58f97 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:42.299589 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-d4ba6531-0245-4aa9-8ee0-793db5f58f97 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cf732a5a623d4cc6a7e3cddb43c417d6', 'request_id': 'req-d4ba6531-0245-4aa9-8ee0-793db5f58f97'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:42.306337 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-d4ba6531-0245-4aa9-8ee0-793db5f58f97 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cf732a5a623d4cc6a7e3cddb43c417d6', 'request_id': 'req-d4ba6531-0245-4aa9-8ee0-793db5f58f97'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:42.311996 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-d4ba6531-0245-4aa9-8ee0-793db5f58f97 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cf732a5a623d4cc6a7e3cddb43c417d6', 'request_id': 'req-d4ba6531-0245-4aa9-8ee0-793db5f58f97'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:42.414028 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-d4ba6531-0245-4aa9-8ee0-793db5f58f97 admin admin] The token cache is being invalidate because project cf732a5a623d4cc6a7e3cddb43c417d6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:50:42.414353 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-d4ba6531-0245-4aa9-8ee0-793db5f58f97 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:42.415012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-d4ba6531-0245-4aa9-8ee0-793db5f58f97 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:42.416573 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1371/2771] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:50:42 2022] DELETE /identity/v3/projects/cf732a5a623d4cc6a7e3cddb43c417d6 => generated 0 bytes in 270 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:50:42.420586 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d942fb-de94-493a-8c25-2e2651fb477b service neutron] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:42.431391 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d942fb-de94-493a-8c25-2e2651fb477b service neutron] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:42.452482 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:50:42.490661 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faca8246-3c0f-43f7-8a11-ba3c460d9b0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:42.491538 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faca8246-3c0f-43f7-8a11-ba3c460d9b0d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:42.491821 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faca8246-3c0f-43f7-8a11-ba3c460d9b0d admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:42.492099 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faca8246-3c0f-43f7-8a11-ba3c460d9b0d admin admin] PATH_INFO: `/v3/users/c812ad0cf5e247b68534446a34f0ec0f` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:42.493295 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faca8246-3c0f-43f7-8a11-ba3c460d9b0d admin admin] RBAC: Authorizing `identity:delete_user(user_id=c812ad0cf5e247b68534446a34f0ec0f)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:42.495347 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faca8246-3c0f-43f7-8a11-ba3c460d9b0d admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:42.876883 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-faca8246-3c0f-43f7-8a11-ba3c460d9b0d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c812ad0cf5e247b68534446a34f0ec0f'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:42.877537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-faca8246-3c0f-43f7-8a11-ba3c460d9b0d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c812ad0cf5e247b68534446a34f0ec0f'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:42.878143 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-faca8246-3c0f-43f7-8a11-ba3c460d9b0d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c812ad0cf5e247b68534446a34f0ec0f'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:42.886397 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-faca8246-3c0f-43f7-8a11-ba3c460d9b0d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c812ad0cf5e247b68534446a34f0ec0f'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:42.899111 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-faca8246-3c0f-43f7-8a11-ba3c460d9b0d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c812ad0cf5e247b68534446a34f0ec0f'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:43.055507 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1401/2772] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:50:42 2022] DELETE /identity/v3/users/c812ad0cf5e247b68534446a34f0ec0f => generated 0 bytes in 635 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:50:43.519521 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ba6531-0245-4aa9-8ee0-793db5f58f97 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:43.521094 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-d4ba6531-0245-4aa9-8ee0-793db5f58f97 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:43.536129 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ba6531-0245-4aa9-8ee0-793db5f58f97 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:43.598176 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfdd10b9-d971-4f2b-aeb7-b76ef9108ce0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:43.599134 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfdd10b9-d971-4f2b-aeb7-b76ef9108ce0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:43.599425 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfdd10b9-d971-4f2b-aeb7-b76ef9108ce0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:43.599746 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfdd10b9-d971-4f2b-aeb7-b76ef9108ce0 admin admin] PATH_INFO: `/v3/projects/a080e8a31f8c47b384a110dee22eb979` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:43.601399 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfdd10b9-d971-4f2b-aeb7-b76ef9108ce0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a080e8a31f8c47b384a110dee22eb979)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:43.603369 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfdd10b9-d971-4f2b-aeb7-b76ef9108ce0 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:43.720916 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:50:43.737489 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-dfdd10b9-d971-4f2b-aeb7-b76ef9108ce0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a080e8a31f8c47b384a110dee22eb979', 'request_id': 'req-dfdd10b9-d971-4f2b-aeb7-b76ef9108ce0'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:43.744378 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-dfdd10b9-d971-4f2b-aeb7-b76ef9108ce0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a080e8a31f8c47b384a110dee22eb979', 'request_id': 'req-dfdd10b9-d971-4f2b-aeb7-b76ef9108ce0'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:43.750145 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-dfdd10b9-d971-4f2b-aeb7-b76ef9108ce0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a080e8a31f8c47b384a110dee22eb979', 'request_id': 'req-dfdd10b9-d971-4f2b-aeb7-b76ef9108ce0'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:43.905368 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-dfdd10b9-d971-4f2b-aeb7-b76ef9108ce0 admin admin] The token cache is being invalidate because project a080e8a31f8c47b384a110dee22eb979 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:50:43.905368 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-dfdd10b9-d971-4f2b-aeb7-b76ef9108ce0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:43.905710 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-dfdd10b9-d971-4f2b-aeb7-b76ef9108ce0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:43.907332 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1372/2773] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:50:43 2022] DELETE /identity/v3/projects/a080e8a31f8c47b384a110dee22eb979 => generated 0 bytes in 388 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:50:43.912388 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d024b283-2ce5-49b6-aee0-595ad201fcfc None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:43.913482 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d024b283-2ce5-49b6-aee0-595ad201fcfc None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:43.913623 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d024b283-2ce5-49b6-aee0-595ad201fcfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:43.932301 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-d024b283-2ce5-49b6-aee0-595ad201fcfc None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:50:43.933542 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-d024b283-2ce5-49b6-aee0-595ad201fcfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:43.997147 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1402/2774] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:50:43 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:50:44.005159 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d0ada1-2163-4aec-b5c8-15d260a7508d None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:44.005582 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d0ada1-2163-4aec-b5c8-15d260a7508d None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:44.005760 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d0ada1-2163-4aec-b5c8-15d260a7508d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:44.025824 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-36d0ada1-2163-4aec-b5c8-15d260a7508d None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:50:44.027082 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-36d0ada1-2163-4aec-b5c8-15d260a7508d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:44.035396 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1373/2775] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:50:44 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:50:44.040486 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d024b283-2ce5-49b6-aee0-595ad201fcfc None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:44.050139 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d024b283-2ce5-49b6-aee0-595ad201fcfc None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:44.054959 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d03a9a2-815f-4e20-84ce-c533eec472b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:44.055613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d03a9a2-815f-4e20-84ce-c533eec472b2 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:44.055897 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d03a9a2-815f-4e20-84ce-c533eec472b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:44.056172 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d03a9a2-815f-4e20-84ce-c533eec472b2 admin admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:44.056906 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d03a9a2-815f-4e20-84ce-c533eec472b2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:50:44.057212 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d03a9a2-815f-4e20-84ce-c533eec472b2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:44.058925 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d03a9a2-815f-4e20-84ce-c533eec472b2 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:44.071045 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1403/2776] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:50:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:50:44.075737 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d0ada1-2163-4aec-b5c8-15d260a7508d None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:44.085137 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d0ada1-2163-4aec-b5c8-15d260a7508d None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:44.089858 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c22cb5-adc2-4bc4-bc30-72b89c61c7a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:44.090543 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c22cb5-adc2-4bc4-bc30-72b89c61c7a7 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:44.090834 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c22cb5-adc2-4bc4-bc30-72b89c61c7a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:44.091115 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c22cb5-adc2-4bc4-bc30-72b89c61c7a7 admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:44.091807 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31c22cb5-adc2-4bc4-bc30-72b89c61c7a7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:44.093702 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31c22cb5-adc2-4bc4-bc30-72b89c61c7a7 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:44.190761 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1374/2777] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:50:44 2022] POST /identity/v3/projects => generated 369 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:50:44.195606 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d03a9a2-815f-4e20-84ce-c533eec472b2 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:44.206566 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d03a9a2-815f-4e20-84ce-c533eec472b2 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:44.234078 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:50:44.266525 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f66f6c0-cf84-476e-82d1-6415805709e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:44.267098 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f66f6c0-cf84-476e-82d1-6415805709e8 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:44.267548 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f66f6c0-cf84-476e-82d1-6415805709e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:44.267838 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f66f6c0-cf84-476e-82d1-6415805709e8 admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:44.268556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f66f6c0-cf84-476e-82d1-6415805709e8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:44.270548 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f66f6c0-cf84-476e-82d1-6415805709e8 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:44.343276 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1404/2778] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:50:44 2022] POST /identity/v3/users => generated 285 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:50:44.347566 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c22cb5-adc2-4bc4-bc30-72b89c61c7a7 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:44.360136 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c22cb5-adc2-4bc4-bc30-72b89c61c7a7 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:44.369187 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc7d529-124d-4414-a96e-7ea524a3137f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:44.369187 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc7d529-124d-4414-a96e-7ea524a3137f admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:44.369187 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc7d529-124d-4414-a96e-7ea524a3137f admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:44.369187 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc7d529-124d-4414-a96e-7ea524a3137f admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:44.369674 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc7d529-124d-4414-a96e-7ea524a3137f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:44.369674 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc7d529-124d-4414-a96e-7ea524a3137f admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:44.379597 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1375/2779] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:50:44 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:50:44.384416 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f66f6c0-cf84-476e-82d1-6415805709e8 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:44.395466 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f66f6c0-cf84-476e-82d1-6415805709e8 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:44.400985 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a07a436-33bb-4c01-9e19-d10b5bb6d3ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:44.401691 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a07a436-33bb-4c01-9e19-d10b5bb6d3ab admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:44.401981 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a07a436-33bb-4c01-9e19-d10b5bb6d3ab admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:44.402264 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a07a436-33bb-4c01-9e19-d10b5bb6d3ab admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:44.403050 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a07a436-33bb-4c01-9e19-d10b5bb6d3ab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:44.404691 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a07a436-33bb-4c01-9e19-d10b5bb6d3ab admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:44.415370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1405/2780] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:50:44 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 12 17:50:44.421958 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc7d529-124d-4414-a96e-7ea524a3137f admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:44.431976 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc7d529-124d-4414-a96e-7ea524a3137f admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:44.436770 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a5cdfd-cc68-4adc-ad70-d5497b4710b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:44.437674 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a5cdfd-cc68-4adc-ad70-d5497b4710b6 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:44.437957 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a5cdfd-cc68-4adc-ad70-d5497b4710b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:44.438256 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a5cdfd-cc68-4adc-ad70-d5497b4710b6 admin admin] PATH_INFO: `/v3/projects/56e99051d71e4db9acf9bdc2aed2a601/users/a121cc36e1db4913a194b036919824e5/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:44.454424 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a5cdfd-cc68-4adc-ad70-d5497b4710b6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=56e99051d71e4db9acf9bdc2aed2a601, user_id=a121cc36e1db4913a194b036919824e5, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:44.457072 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a5cdfd-cc68-4adc-ad70-d5497b4710b6 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:44.514583 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1376/2781] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:50:44 2022] PUT /identity/v3/projects/56e99051d71e4db9acf9bdc2aed2a601/users/a121cc36e1db4913a194b036919824e5/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 93 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:50:44.525288 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a24780-aa57-4913-90a8-aade37adbb81 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:44.525521 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a24780-aa57-4913-90a8-aade37adbb81 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:44.525818 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a24780-aa57-4913-90a8-aade37adbb81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:44.552523 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-90a24780-aa57-4913-90a8-aade37adbb81 None None] MFA Rules not processed for user `a121cc36e1db4913a194b036919824e5`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:50:44.554350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-90a24780-aa57-4913-90a8-aade37adbb81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:44.625358 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1406/2782] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:50:44 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:50:44.630418 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a5cdfd-cc68-4adc-ad70-d5497b4710b6 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:44.699004 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a5cdfd-cc68-4adc-ad70-d5497b4710b6 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:44.704068 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b698192-3a8f-42da-9228-0681218bcce7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:44.704846 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b698192-3a8f-42da-9228-0681218bcce7 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:44.705160 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b698192-3a8f-42da-9228-0681218bcce7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:44.705468 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b698192-3a8f-42da-9228-0681218bcce7 admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:44.706157 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b698192-3a8f-42da-9228-0681218bcce7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:44.708087 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b698192-3a8f-42da-9228-0681218bcce7 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:44.859056 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1377/2783] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:50:44 2022] POST /identity/v3/projects => generated 369 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:50:44.863147 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a24780-aa57-4913-90a8-aade37adbb81 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:44.875256 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a24780-aa57-4913-90a8-aade37adbb81 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:44.942966 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53cb6345-9605-49d8-9e0d-2ffe6a595100 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:44.943669 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53cb6345-9605-49d8-9e0d-2ffe6a595100 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:44.943963 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53cb6345-9605-49d8-9e0d-2ffe6a595100 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:44.944246 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53cb6345-9605-49d8-9e0d-2ffe6a595100 admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:44.944938 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53cb6345-9605-49d8-9e0d-2ffe6a595100 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:44.946907 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53cb6345-9605-49d8-9e0d-2ffe6a595100 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:45.057977 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1407/2784] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:50:44 2022] POST /identity/v3/users => generated 291 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:50:45.062148 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b698192-3a8f-42da-9228-0681218bcce7 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:45.072732 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b698192-3a8f-42da-9228-0681218bcce7 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:45.077467 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb9f9d3-4e3a-42e6-9c2f-e46d8fcd8af2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:45.078124 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb9f9d3-4e3a-42e6-9c2f-e46d8fcd8af2 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:45.078416 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb9f9d3-4e3a-42e6-9c2f-e46d8fcd8af2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:45.078730 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb9f9d3-4e3a-42e6-9c2f-e46d8fcd8af2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:45.079471 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deb9f9d3-4e3a-42e6-9c2f-e46d8fcd8af2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:45.081115 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deb9f9d3-4e3a-42e6-9c2f-e46d8fcd8af2 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:45.092082 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1378/2785] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:50:45 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:50:45.096235 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53cb6345-9605-49d8-9e0d-2ffe6a595100 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:45.106950 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53cb6345-9605-49d8-9e0d-2ffe6a595100 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:45.111590 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36264445-1083-41ef-ab93-1a408c343f7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:45.112402 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36264445-1083-41ef-ab93-1a408c343f7e admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:45.112687 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36264445-1083-41ef-ab93-1a408c343f7e admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:45.112965 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36264445-1083-41ef-ab93-1a408c343f7e admin admin] PATH_INFO: `/v3/projects/37223edfd7154ac384f76bb7ed156853/users/81dd5e7f518c4169b746b316251f4447/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:45.128893 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36264445-1083-41ef-ab93-1a408c343f7e admin admin] RBAC: Authorizing `identity:create_grant(project_id=37223edfd7154ac384f76bb7ed156853, user_id=81dd5e7f518c4169b746b316251f4447, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:45.131665 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36264445-1083-41ef-ab93-1a408c343f7e admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:45.198483 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1408/2786] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:50:45 2022] PUT /identity/v3/projects/37223edfd7154ac384f76bb7ed156853/users/81dd5e7f518c4169b746b316251f4447/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:50:45.202944 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb9f9d3-4e3a-42e6-9c2f-e46d8fcd8af2 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:45.213765 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb9f9d3-4e3a-42e6-9c2f-e46d8fcd8af2 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:45.238860 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:50:45.278853 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f39cc92-4019-4995-850b-3886a27f6613 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:45.279534 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f39cc92-4019-4995-850b-3886a27f6613 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:45.279822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f39cc92-4019-4995-850b-3886a27f6613 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:45.280120 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f39cc92-4019-4995-850b-3886a27f6613 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:45.280911 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f39cc92-4019-4995-850b-3886a27f6613 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:45.282692 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f39cc92-4019-4995-850b-3886a27f6613 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:45.295061 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1379/2787] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:50:45 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:50:45.299529 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36264445-1083-41ef-ab93-1a408c343f7e admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:45.305205 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:50:45.310757 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36264445-1083-41ef-ab93-1a408c343f7e admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:45.316012 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14051e2-5345-412c-bde5-21d2649cc097 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:45.317005 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14051e2-5345-412c-bde5-21d2649cc097 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:45.317459 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14051e2-5345-412c-bde5-21d2649cc097 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:45.317736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14051e2-5345-412c-bde5-21d2649cc097 admin admin] PATH_INFO: `/v3/projects/37223edfd7154ac384f76bb7ed156853/users/81dd5e7f518c4169b746b316251f4447/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:45.320251 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f14051e2-5345-412c-bde5-21d2649cc097 admin admin] RBAC: Authorizing `identity:create_grant(project_id=37223edfd7154ac384f76bb7ed156853, user_id=81dd5e7f518c4169b746b316251f4447, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:45.323190 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f14051e2-5345-412c-bde5-21d2649cc097 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:45.335277 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1409/2788] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:50:45 2022] PUT /identity/v3/projects/37223edfd7154ac384f76bb7ed156853/users/81dd5e7f518c4169b746b316251f4447/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:50:45.346543 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1ffb2c-68d0-4779-9662-d5139a050c92 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:45.346825 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1ffb2c-68d0-4779-9662-d5139a050c92 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:45.347120 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1ffb2c-68d0-4779-9662-d5139a050c92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:45.376461 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-cf1ffb2c-68d0-4779-9662-d5139a050c92 None None] MFA Rules not processed for user `81dd5e7f518c4169b746b316251f4447`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:50:45.377654 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-cf1ffb2c-68d0-4779-9662-d5139a050c92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:45.447560 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1380/2789] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:50:45 2022] POST /identity/v3/auth/tokens => generated 3303 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:50:45.460099 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14051e2-5345-412c-bde5-21d2649cc097 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:45.462129 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-f14051e2-5345-412c-bde5-21d2649cc097 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:45.522733 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14051e2-5345-412c-bde5-21d2649cc097 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:45.526489 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79bfa11b-1465-445a-a14c-d1f3da6d61c7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:45.527142 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79bfa11b-1465-445a-a14c-d1f3da6d61c7 service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:45.527432 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79bfa11b-1465-445a-a14c-d1f3da6d61c7 service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:45.527720 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79bfa11b-1465-445a-a14c-d1f3da6d61c7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:45.537270 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79bfa11b-1465-445a-a14c-d1f3da6d61c7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:45.539454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79bfa11b-1465-445a-a14c-d1f3da6d61c7 service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:45.549609 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1410/2790] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:50:45 2022] GET /identity/v3/auth/tokens => generated 3303 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:50:48.478065 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1ffb2c-68d0-4779-9662-d5139a050c92 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:48.543513 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1ffb2c-68d0-4779-9662-d5139a050c92 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:48.547998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534b4d89-d246-4bfc-83d4-f19715a992dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:48.548866 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534b4d89-d246-4bfc-83d4-f19715a992dd admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:48.549154 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534b4d89-d246-4bfc-83d4-f19715a992dd admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:48.549436 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534b4d89-d246-4bfc-83d4-f19715a992dd admin admin] PATH_INFO: `/v3/users/a121cc36e1db4913a194b036919824e5` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:48.550657 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534b4d89-d246-4bfc-83d4-f19715a992dd admin admin] RBAC: Authorizing `identity:delete_user(user_id=a121cc36e1db4913a194b036919824e5)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:48.552561 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534b4d89-d246-4bfc-83d4-f19715a992dd admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:48.747104 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-534b4d89-d246-4bfc-83d4-f19715a992dd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a121cc36e1db4913a194b036919824e5'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:48.748811 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:50:48.804648 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-534b4d89-d246-4bfc-83d4-f19715a992dd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a121cc36e1db4913a194b036919824e5'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:48.805327 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-534b4d89-d246-4bfc-83d4-f19715a992dd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a121cc36e1db4913a194b036919824e5'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:48.812832 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-534b4d89-d246-4bfc-83d4-f19715a992dd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a121cc36e1db4913a194b036919824e5'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:48.826694 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-534b4d89-d246-4bfc-83d4-f19715a992dd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a121cc36e1db4913a194b036919824e5'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:48.832788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1381/2791] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:50:48 2022] DELETE /identity/v3/users/a121cc36e1db4913a194b036919824e5 => generated 0 bytes in 355 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:50:48.837957 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d04eac7-2b02-40d7-8d3e-885aa3db6e5e None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:48.838492 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d04eac7-2b02-40d7-8d3e-885aa3db6e5e None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:48.838615 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d04eac7-2b02-40d7-8d3e-885aa3db6e5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:48.856694 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-5d04eac7-2b02-40d7-8d3e-885aa3db6e5e None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:50:48.857932 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-5d04eac7-2b02-40d7-8d3e-885aa3db6e5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:48.918804 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1411/2792] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:50:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:50:48.928668 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534b4d89-d246-4bfc-83d4-f19715a992dd admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:48.930134 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-534b4d89-d246-4bfc-83d4-f19715a992dd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:48.990050 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534b4d89-d246-4bfc-83d4-f19715a992dd admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:48.994496 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26204b98-bb49-4fdd-ad65-aa4ebf2506c7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:48.995144 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26204b98-bb49-4fdd-ad65-aa4ebf2506c7 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:48.995433 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26204b98-bb49-4fdd-ad65-aa4ebf2506c7 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:48.995715 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26204b98-bb49-4fdd-ad65-aa4ebf2506c7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:49.004824 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26204b98-bb49-4fdd-ad65-aa4ebf2506c7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:49.006363 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26204b98-bb49-4fdd-ad65-aa4ebf2506c7 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:49.016327 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1382/2793] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:50:48 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:50:49.409494 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d04eac7-2b02-40d7-8d3e-885aa3db6e5e None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:49.410962 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-5d04eac7-2b02-40d7-8d3e-885aa3db6e5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:49.424252 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d04eac7-2b02-40d7-8d3e-885aa3db6e5e None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:49.429968 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d874b0-4f43-4159-8fdd-cd849283d1bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:49.430916 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d874b0-4f43-4159-8fdd-cd849283d1bb admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:49.431207 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d874b0-4f43-4159-8fdd-cd849283d1bb admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:49.437254 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d874b0-4f43-4159-8fdd-cd849283d1bb admin admin] PATH_INFO: `/v3/projects/56e99051d71e4db9acf9bdc2aed2a601` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:49.437254 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d874b0-4f43-4159-8fdd-cd849283d1bb admin admin] RBAC: Authorizing `identity:delete_project(project_id=56e99051d71e4db9acf9bdc2aed2a601)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:49.437254 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d874b0-4f43-4159-8fdd-cd849283d1bb admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:49.534267 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-90d874b0-4f43-4159-8fdd-cd849283d1bb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '56e99051d71e4db9acf9bdc2aed2a601', 'request_id': 'req-90d874b0-4f43-4159-8fdd-cd849283d1bb'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:49.540369 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-90d874b0-4f43-4159-8fdd-cd849283d1bb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '56e99051d71e4db9acf9bdc2aed2a601', 'request_id': 'req-90d874b0-4f43-4159-8fdd-cd849283d1bb'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:49.604678 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-90d874b0-4f43-4159-8fdd-cd849283d1bb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '56e99051d71e4db9acf9bdc2aed2a601', 'request_id': 'req-90d874b0-4f43-4159-8fdd-cd849283d1bb'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:49.617481 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-90d874b0-4f43-4159-8fdd-cd849283d1bb admin admin] The token cache is being invalidate because project 56e99051d71e4db9acf9bdc2aed2a601 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:50:49.617714 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-90d874b0-4f43-4159-8fdd-cd849283d1bb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:49.618368 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-90d874b0-4f43-4159-8fdd-cd849283d1bb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:49.619984 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1412/2794] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:50:49 2022] DELETE /identity/v3/projects/56e99051d71e4db9acf9bdc2aed2a601 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:50:49.624004 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26204b98-bb49-4fdd-ad65-aa4ebf2506c7 service neutron] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:49.634619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26204b98-bb49-4fdd-ad65-aa4ebf2506c7 service neutron] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:49.691684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823f2663-7cf6-49f2-a269-c0681f3044b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:49.692591 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823f2663-7cf6-49f2-a269-c0681f3044b8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:49.692884 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823f2663-7cf6-49f2-a269-c0681f3044b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:49.693171 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823f2663-7cf6-49f2-a269-c0681f3044b8 admin admin] PATH_INFO: `/v3/users/81dd5e7f518c4169b746b316251f4447` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:49.694382 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-823f2663-7cf6-49f2-a269-c0681f3044b8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=81dd5e7f518c4169b746b316251f4447)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:49.696323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-823f2663-7cf6-49f2-a269-c0681f3044b8 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:49.847395 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-823f2663-7cf6-49f2-a269-c0681f3044b8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '81dd5e7f518c4169b746b316251f4447'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:49.902177 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-823f2663-7cf6-49f2-a269-c0681f3044b8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '81dd5e7f518c4169b746b316251f4447'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:49.902749 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-823f2663-7cf6-49f2-a269-c0681f3044b8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '81dd5e7f518c4169b746b316251f4447'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:49.910370 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-823f2663-7cf6-49f2-a269-c0681f3044b8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '81dd5e7f518c4169b746b316251f4447'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:49.916348 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:INSERT=2,keystone:DELETE=12,keystone:SELECT=86 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:50:49.923150 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-823f2663-7cf6-49f2-a269-c0681f3044b8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '81dd5e7f518c4169b746b316251f4447'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:49.929258 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1383/2795] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:50:49 2022] DELETE /identity/v3/users/81dd5e7f518c4169b746b316251f4447 => generated 0 bytes in 306 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:50:50.364049 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d874b0-4f43-4159-8fdd-cd849283d1bb admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:50.365495 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-90d874b0-4f43-4159-8fdd-cd849283d1bb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:50.383994 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d874b0-4f43-4159-8fdd-cd849283d1bb admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:50.398384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:50:50.443186 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75934496-371f-4f6a-b264-ff8906e1b241 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:50.444540 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75934496-371f-4f6a-b264-ff8906e1b241 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:50.444833 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75934496-371f-4f6a-b264-ff8906e1b241 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:50.445112 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75934496-371f-4f6a-b264-ff8906e1b241 admin admin] PATH_INFO: `/v3/projects/37223edfd7154ac384f76bb7ed156853` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:50.446380 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75934496-371f-4f6a-b264-ff8906e1b241 admin admin] RBAC: Authorizing `identity:delete_project(project_id=37223edfd7154ac384f76bb7ed156853)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:50.448317 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75934496-371f-4f6a-b264-ff8906e1b241 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:50.592767 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-75934496-371f-4f6a-b264-ff8906e1b241 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '37223edfd7154ac384f76bb7ed156853', 'request_id': 'req-75934496-371f-4f6a-b264-ff8906e1b241'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:50.598332 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-75934496-371f-4f6a-b264-ff8906e1b241 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '37223edfd7154ac384f76bb7ed156853', 'request_id': 'req-75934496-371f-4f6a-b264-ff8906e1b241'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:50.668880 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-75934496-371f-4f6a-b264-ff8906e1b241 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '37223edfd7154ac384f76bb7ed156853', 'request_id': 'req-75934496-371f-4f6a-b264-ff8906e1b241'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:50.681967 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-75934496-371f-4f6a-b264-ff8906e1b241 admin admin] The token cache is being invalidate because project 37223edfd7154ac384f76bb7ed156853 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:50:50.682268 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-75934496-371f-4f6a-b264-ff8906e1b241 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:50.683053 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-75934496-371f-4f6a-b264-ff8906e1b241 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:50.684755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1413/2796] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:50:50 2022] DELETE /identity/v3/projects/37223edfd7154ac384f76bb7ed156853 => generated 0 bytes in 321 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:50:50.690103 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ec3b77-6944-419e-90ce-9715325ca143 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:50.693395 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ec3b77-6944-419e-90ce-9715325ca143 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:50.693395 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ec3b77-6944-419e-90ce-9715325ca143 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:50.710731 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-d6ec3b77-6944-419e-90ce-9715325ca143 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:50:50.711956 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-d6ec3b77-6944-419e-90ce-9715325ca143 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:50.774281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1384/2797] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:50:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:50:50.782287 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b13d710-09bd-4b32-b158-04bcfebbecf0 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:50.782625 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b13d710-09bd-4b32-b158-04bcfebbecf0 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:50.782914 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b13d710-09bd-4b32-b158-04bcfebbecf0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:50.801680 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-7b13d710-09bd-4b32-b158-04bcfebbecf0 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:50:50.802944 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-7b13d710-09bd-4b32-b158-04bcfebbecf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:50.811937 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1414/2798] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:50:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:50:50.817319 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ec3b77-6944-419e-90ce-9715325ca143 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:50.828231 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ec3b77-6944-419e-90ce-9715325ca143 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:50.832848 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ed018c-d923-476e-be7a-e690051a1603 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:50.833514 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ed018c-d923-476e-be7a-e690051a1603 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:50.833800 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ed018c-d923-476e-be7a-e690051a1603 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:50.834080 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ed018c-d923-476e-be7a-e690051a1603 admin admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:50.835029 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ed018c-d923-476e-be7a-e690051a1603 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:50:50.835343 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ed018c-d923-476e-be7a-e690051a1603 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:50.836936 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ed018c-d923-476e-be7a-e690051a1603 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:50.848765 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1385/2799] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:50:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:50:50.853643 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b13d710-09bd-4b32-b158-04bcfebbecf0 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:50.863278 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b13d710-09bd-4b32-b158-04bcfebbecf0 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:50.868906 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420e2ecb-6ec5-4c6e-92ec-03c22286662a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:50.869575 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420e2ecb-6ec5-4c6e-92ec-03c22286662a admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:50.869921 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420e2ecb-6ec5-4c6e-92ec-03c22286662a admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:50.870141 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420e2ecb-6ec5-4c6e-92ec-03c22286662a admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:50.870880 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-420e2ecb-6ec5-4c6e-92ec-03c22286662a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:50.872795 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-420e2ecb-6ec5-4c6e-92ec-03c22286662a admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:50.982089 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1415/2800] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:50:50 2022] POST /identity/v3/projects => generated 381 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:50:50.986421 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ed018c-d923-476e-be7a-e690051a1603 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:50.997383 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ed018c-d923-476e-be7a-e690051a1603 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:51.056579 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4416ff9f-9eae-4939-812f-29dcdaa39cfe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:51.057268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4416ff9f-9eae-4939-812f-29dcdaa39cfe admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:51.057562 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4416ff9f-9eae-4939-812f-29dcdaa39cfe admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:51.057847 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4416ff9f-9eae-4939-812f-29dcdaa39cfe admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:51.058597 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4416ff9f-9eae-4939-812f-29dcdaa39cfe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:51.060527 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4416ff9f-9eae-4939-812f-29dcdaa39cfe admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:51.165513 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1386/2801] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:50:50 2022] POST /identity/v3/users => generated 291 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:50:51.169902 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420e2ecb-6ec5-4c6e-92ec-03c22286662a admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:51.184298 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420e2ecb-6ec5-4c6e-92ec-03c22286662a admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:51.189221 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f4761dc-8d24-4161-8aef-4cef49e668fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:51.190144 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4761dc-8d24-4161-8aef-4cef49e668fc admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:51.190451 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4761dc-8d24-4161-8aef-4cef49e668fc admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:51.190917 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4761dc-8d24-4161-8aef-4cef49e668fc admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:51.191680 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f4761dc-8d24-4161-8aef-4cef49e668fc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:51.193331 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f4761dc-8d24-4161-8aef-4cef49e668fc admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:51.204703 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1416/2802] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:50:51 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:50:51.210134 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4416ff9f-9eae-4939-812f-29dcdaa39cfe admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:51.222630 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4416ff9f-9eae-4939-812f-29dcdaa39cfe admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:51.228380 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e3abdb-14fe-4bd3-9d2c-161ef9bad104 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:51.229049 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e3abdb-14fe-4bd3-9d2c-161ef9bad104 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:51.229337 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e3abdb-14fe-4bd3-9d2c-161ef9bad104 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:51.229614 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e3abdb-14fe-4bd3-9d2c-161ef9bad104 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:51.230346 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e3abdb-14fe-4bd3-9d2c-161ef9bad104 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:51.232163 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e3abdb-14fe-4bd3-9d2c-161ef9bad104 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:51.244037 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1387/2803] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:50:51 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 12 17:50:51.248312 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f4761dc-8d24-4161-8aef-4cef49e668fc admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:51.259692 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f4761dc-8d24-4161-8aef-4cef49e668fc admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:51.266654 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0e2eba-1099-4eb0-9357-4c59f915e06b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:51.267557 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0e2eba-1099-4eb0-9357-4c59f915e06b admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:51.267848 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0e2eba-1099-4eb0-9357-4c59f915e06b admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:51.268127 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0e2eba-1099-4eb0-9357-4c59f915e06b admin admin] PATH_INFO: `/v3/projects/3643b022914b456c904b93eca768b381/users/00ca6fdff81b474285d9f54f4b2255c3/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:51.291531 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb0e2eba-1099-4eb0-9357-4c59f915e06b admin admin] RBAC: Authorizing `identity:create_grant(project_id=3643b022914b456c904b93eca768b381, user_id=00ca6fdff81b474285d9f54f4b2255c3, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:51.294205 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb0e2eba-1099-4eb0-9357-4c59f915e06b admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:51.369282 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1417/2804] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:50:51 2022] PUT /identity/v3/projects/3643b022914b456c904b93eca768b381/users/00ca6fdff81b474285d9f54f4b2255c3/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:50:51.381867 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7bcd2f-aee2-4fca-902b-301a90b26b14 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:51.382201 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7bcd2f-aee2-4fca-902b-301a90b26b14 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:51.382509 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7bcd2f-aee2-4fca-902b-301a90b26b14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:51.413699 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-5a7bcd2f-aee2-4fca-902b-301a90b26b14 None None] MFA Rules not processed for user `00ca6fdff81b474285d9f54f4b2255c3`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:50:51.419532 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-5a7bcd2f-aee2-4fca-902b-301a90b26b14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:51.468729 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:50:51.502904 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1388/2805] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:50:51 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:50:51.508323 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0e2eba-1099-4eb0-9357-4c59f915e06b admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:51.559829 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:50:51.593585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0e2eba-1099-4eb0-9357-4c59f915e06b admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:51.599359 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9eafa40-4979-4b81-8b62-3a5c153450ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:51.600272 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9eafa40-4979-4b81-8b62-3a5c153450ec admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:51.602505 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9eafa40-4979-4b81-8b62-3a5c153450ec admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:51.602505 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9eafa40-4979-4b81-8b62-3a5c153450ec admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:51.602505 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9eafa40-4979-4b81-8b62-3a5c153450ec admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:51.603690 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9eafa40-4979-4b81-8b62-3a5c153450ec admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:51.716750 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1418/2806] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:50:51 2022] POST /identity/v3/projects => generated 381 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:50:51.720859 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a7bcd2f-aee2-4fca-902b-301a90b26b14 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:51.737009 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a7bcd2f-aee2-4fca-902b-301a90b26b14 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:51.799152 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-934699bb-0e79-4af9-a29f-21105886bb07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:51.799743 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934699bb-0e79-4af9-a29f-21105886bb07 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:51.800032 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934699bb-0e79-4af9-a29f-21105886bb07 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:51.800313 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934699bb-0e79-4af9-a29f-21105886bb07 admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:51.801023 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-934699bb-0e79-4af9-a29f-21105886bb07 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:51.802969 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-934699bb-0e79-4af9-a29f-21105886bb07 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:51.900401 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1389/2807] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:50:51 2022] POST /identity/v3/users => generated 297 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:50:51.904680 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9eafa40-4979-4b81-8b62-3a5c153450ec admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:51.915450 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9eafa40-4979-4b81-8b62-3a5c153450ec admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:51.925274 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb129b6-70d1-4a95-ad2a-84aa5fdde0d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:51.925942 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb129b6-70d1-4a95-ad2a-84aa5fdde0d9 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:51.926232 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb129b6-70d1-4a95-ad2a-84aa5fdde0d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:51.926539 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb129b6-70d1-4a95-ad2a-84aa5fdde0d9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:51.927308 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eb129b6-70d1-4a95-ad2a-84aa5fdde0d9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:51.928940 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eb129b6-70d1-4a95-ad2a-84aa5fdde0d9 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:51.941563 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1419/2808] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:50:51 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:50:51.946964 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-934699bb-0e79-4af9-a29f-21105886bb07 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:51.957606 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-934699bb-0e79-4af9-a29f-21105886bb07 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:51.963098 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4d3c14-ba17-4492-97b9-dcb80d6c0f82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:51.964009 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4d3c14-ba17-4492-97b9-dcb80d6c0f82 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:51.964324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4d3c14-ba17-4492-97b9-dcb80d6c0f82 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:51.964609 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4d3c14-ba17-4492-97b9-dcb80d6c0f82 admin admin] PATH_INFO: `/v3/projects/45dd29a3dd7645699789116580436b62/users/4455d52882344e3e9278b2cbc2cf823b/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:51.981643 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d4d3c14-ba17-4492-97b9-dcb80d6c0f82 admin admin] RBAC: Authorizing `identity:create_grant(project_id=45dd29a3dd7645699789116580436b62, user_id=4455d52882344e3e9278b2cbc2cf823b, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:51.984448 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d4d3c14-ba17-4492-97b9-dcb80d6c0f82 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:52.068159 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1390/2809] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:50:51 2022] PUT /identity/v3/projects/45dd29a3dd7645699789116580436b62/users/4455d52882344e3e9278b2cbc2cf823b/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:50:52.073022 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb129b6-70d1-4a95-ad2a-84aa5fdde0d9 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:52.083542 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb129b6-70d1-4a95-ad2a-84aa5fdde0d9 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:52.142379 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bffcf5e2-5fd8-4a6d-81a1-854399fd122a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:52.143089 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bffcf5e2-5fd8-4a6d-81a1-854399fd122a admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:52.143378 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bffcf5e2-5fd8-4a6d-81a1-854399fd122a admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:52.143666 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bffcf5e2-5fd8-4a6d-81a1-854399fd122a admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:52.144478 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bffcf5e2-5fd8-4a6d-81a1-854399fd122a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:52.146093 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bffcf5e2-5fd8-4a6d-81a1-854399fd122a admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:52.156859 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1420/2810] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:50:52 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 12 17:50:52.161107 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4d3c14-ba17-4492-97b9-dcb80d6c0f82 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:52.173623 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4d3c14-ba17-4492-97b9-dcb80d6c0f82 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:52.179055 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030da7ad-1095-486d-b92e-fff4a68fe25b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:52.187541 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030da7ad-1095-486d-b92e-fff4a68fe25b admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:52.187541 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030da7ad-1095-486d-b92e-fff4a68fe25b admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:52.187541 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030da7ad-1095-486d-b92e-fff4a68fe25b admin admin] PATH_INFO: `/v3/projects/45dd29a3dd7645699789116580436b62/users/4455d52882344e3e9278b2cbc2cf823b/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:52.187541 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030da7ad-1095-486d-b92e-fff4a68fe25b admin admin] RBAC: Authorizing `identity:create_grant(project_id=45dd29a3dd7645699789116580436b62, user_id=4455d52882344e3e9278b2cbc2cf823b, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:52.187541 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030da7ad-1095-486d-b92e-fff4a68fe25b admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:52.197144 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1391/2811] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:50:52 2022] PUT /identity/v3/projects/45dd29a3dd7645699789116580436b62/users/4455d52882344e3e9278b2cbc2cf823b/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:50:52.261550 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38d10a2-6824-4570-a7ea-17c655117d6b None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:52.261844 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38d10a2-6824-4570-a7ea-17c655117d6b None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:52.262128 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38d10a2-6824-4570-a7ea-17c655117d6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:52.289122 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-a38d10a2-6824-4570-a7ea-17c655117d6b None None] MFA Rules not processed for user `4455d52882344e3e9278b2cbc2cf823b`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:50:52.290980 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-a38d10a2-6824-4570-a7ea-17c655117d6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:52.366421 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1421/2812] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:50:52 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:50:52.378303 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030da7ad-1095-486d-b92e-fff4a68fe25b admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:52.380055 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-030da7ad-1095-486d-b92e-fff4a68fe25b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:52.456405 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030da7ad-1095-486d-b92e-fff4a68fe25b admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:52.460123 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0597fe3-103b-40e2-937e-e4e50068b613 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:52.460768 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0597fe3-103b-40e2-937e-e4e50068b613 service cinder] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:52.461057 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0597fe3-103b-40e2-937e-e4e50068b613 service cinder] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:52.461382 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0597fe3-103b-40e2-937e-e4e50068b613 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:52.471324 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0597fe3-103b-40e2-937e-e4e50068b613 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:52.472842 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0597fe3-103b-40e2-937e-e4e50068b613 service cinder] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:52.483436 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1392/2813] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:50:52 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:50:52.642408 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38d10a2-6824-4570-a7ea-17c655117d6b None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:52.652286 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38d10a2-6824-4570-a7ea-17c655117d6b None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:52.656210 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2176ec4a-d642-442b-a69a-9b76e263f7d1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:52.656855 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2176ec4a-d642-442b-a69a-9b76e263f7d1 service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:52.657139 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2176ec4a-d642-442b-a69a-9b76e263f7d1 service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:52.657416 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2176ec4a-d642-442b-a69a-9b76e263f7d1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:52.694854 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:50:52.718366 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2176ec4a-d642-442b-a69a-9b76e263f7d1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:52.719979 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2176ec4a-d642-442b-a69a-9b76e263f7d1 service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:52.729865 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1422/2814] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:50:52 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:50:53.295586 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0597fe3-103b-40e2-937e-e4e50068b613 service cinder] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:53.340458 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:50:53.361471 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0597fe3-103b-40e2-937e-e4e50068b613 service cinder] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:53.366268 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86152750-9804-497e-852a-f1fa0dc8822b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:53.367550 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86152750-9804-497e-852a-f1fa0dc8822b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:53.367968 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86152750-9804-497e-852a-f1fa0dc8822b admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:53.368373 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86152750-9804-497e-852a-f1fa0dc8822b admin admin] PATH_INFO: `/v3/users/00ca6fdff81b474285d9f54f4b2255c3` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:53.369649 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86152750-9804-497e-852a-f1fa0dc8822b admin admin] RBAC: Authorizing `identity:delete_user(user_id=00ca6fdff81b474285d9f54f4b2255c3)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:53.371610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86152750-9804-497e-852a-f1fa0dc8822b admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:53.595976 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-86152750-9804-497e-852a-f1fa0dc8822b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '00ca6fdff81b474285d9f54f4b2255c3'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:53.691024 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-86152750-9804-497e-852a-f1fa0dc8822b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '00ca6fdff81b474285d9f54f4b2255c3'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:53.691593 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-86152750-9804-497e-852a-f1fa0dc8822b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '00ca6fdff81b474285d9f54f4b2255c3'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:53.699703 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-86152750-9804-497e-852a-f1fa0dc8822b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '00ca6fdff81b474285d9f54f4b2255c3'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:53.711832 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-86152750-9804-497e-852a-f1fa0dc8822b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '00ca6fdff81b474285d9f54f4b2255c3'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:53.717863 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1393/2815] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:50:53 2022] DELETE /identity/v3/users/00ca6fdff81b474285d9f54f4b2255c3 => generated 0 bytes in 423 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:50:53.722591 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ac87c7-48c1-4a42-bdb4-86692250d1e8 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:53.723143 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ac87c7-48c1-4a42-bdb4-86692250d1e8 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:53.723273 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ac87c7-48c1-4a42-bdb4-86692250d1e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:53.741322 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-a2ac87c7-48c1-4a42-bdb4-86692250d1e8 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:50:53.742706 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-a2ac87c7-48c1-4a42-bdb4-86692250d1e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:53.804075 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1423/2816] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:50:53 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:50:53.814155 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86152750-9804-497e-852a-f1fa0dc8822b admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:53.815885 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-86152750-9804-497e-852a-f1fa0dc8822b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:53.874167 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86152750-9804-497e-852a-f1fa0dc8822b admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:53.877702 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bfa17dc-ab5c-4c76-b876-e4623b30f08e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:53.878349 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfa17dc-ab5c-4c76-b876-e4623b30f08e service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:53.878669 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfa17dc-ab5c-4c76-b876-e4623b30f08e service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:53.878955 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bfa17dc-ab5c-4c76-b876-e4623b30f08e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:53.888416 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bfa17dc-ab5c-4c76-b876-e4623b30f08e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:53.889908 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bfa17dc-ab5c-4c76-b876-e4623b30f08e service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:53.899989 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1394/2817] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:50:53 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:50:54.296215 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ac87c7-48c1-4a42-bdb4-86692250d1e8 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:54.298570 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-a2ac87c7-48c1-4a42-bdb4-86692250d1e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:54.315291 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ac87c7-48c1-4a42-bdb4-86692250d1e8 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:54.321130 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2925c011-e761-4d08-a99d-a8625b60a619 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:54.322064 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2925c011-e761-4d08-a99d-a8625b60a619 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:54.322349 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2925c011-e761-4d08-a99d-a8625b60a619 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:54.322654 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2925c011-e761-4d08-a99d-a8625b60a619 admin admin] PATH_INFO: `/v3/projects/3643b022914b456c904b93eca768b381` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:54.323931 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2925c011-e761-4d08-a99d-a8625b60a619 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3643b022914b456c904b93eca768b381)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:54.325848 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2925c011-e761-4d08-a99d-a8625b60a619 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:54.431658 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-2925c011-e761-4d08-a99d-a8625b60a619 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3643b022914b456c904b93eca768b381', 'request_id': 'req-2925c011-e761-4d08-a99d-a8625b60a619'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:54.437432 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-2925c011-e761-4d08-a99d-a8625b60a619 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3643b022914b456c904b93eca768b381', 'request_id': 'req-2925c011-e761-4d08-a99d-a8625b60a619'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:54.523207 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-2925c011-e761-4d08-a99d-a8625b60a619 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3643b022914b456c904b93eca768b381', 'request_id': 'req-2925c011-e761-4d08-a99d-a8625b60a619'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:54.535008 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-2925c011-e761-4d08-a99d-a8625b60a619 admin admin] The token cache is being invalidate because project 3643b022914b456c904b93eca768b381 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:50:54.535497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-2925c011-e761-4d08-a99d-a8625b60a619 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:54.536164 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-2925c011-e761-4d08-a99d-a8625b60a619 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:54.538974 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1424/2818] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:50:54 2022] DELETE /identity/v3/projects/3643b022914b456c904b93eca768b381 => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:50:54.542931 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bfa17dc-ab5c-4c76-b876-e4623b30f08e service neutron] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:54.552758 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bfa17dc-ab5c-4c76-b876-e4623b30f08e service neutron] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:54.633917 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce92a18-9701-479a-951e-32ae61807ea2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:54.634886 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce92a18-9701-479a-951e-32ae61807ea2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:54.635200 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce92a18-9701-479a-951e-32ae61807ea2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:54.635485 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce92a18-9701-479a-951e-32ae61807ea2 admin admin] PATH_INFO: `/v3/users/4455d52882344e3e9278b2cbc2cf823b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:54.636687 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce92a18-9701-479a-951e-32ae61807ea2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4455d52882344e3e9278b2cbc2cf823b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:54.638643 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce92a18-9701-479a-951e-32ae61807ea2 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:54.640291 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:50:54.864075 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-3ce92a18-9701-479a-951e-32ae61807ea2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4455d52882344e3e9278b2cbc2cf823b'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:54.959193 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-3ce92a18-9701-479a-951e-32ae61807ea2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4455d52882344e3e9278b2cbc2cf823b'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:54.960407 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-3ce92a18-9701-479a-951e-32ae61807ea2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4455d52882344e3e9278b2cbc2cf823b'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:54.967967 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-3ce92a18-9701-479a-951e-32ae61807ea2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4455d52882344e3e9278b2cbc2cf823b'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:54.980236 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-3ce92a18-9701-479a-951e-32ae61807ea2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4455d52882344e3e9278b2cbc2cf823b'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:54.987333 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1395/2819] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:50:54 2022] DELETE /identity/v3/users/4455d52882344e3e9278b2cbc2cf823b => generated 0 bytes in 444 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:50:55.425332 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2925c011-e761-4d08-a99d-a8625b60a619 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:55.427134 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-2925c011-e761-4d08-a99d-a8625b60a619 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:55.441843 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2925c011-e761-4d08-a99d-a8625b60a619 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:55.503952 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36440c36-8268-4a56-b402-3a4438e431c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:55.504854 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36440c36-8268-4a56-b402-3a4438e431c0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:55.505137 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36440c36-8268-4a56-b402-3a4438e431c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:55.505410 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36440c36-8268-4a56-b402-3a4438e431c0 admin admin] PATH_INFO: `/v3/projects/45dd29a3dd7645699789116580436b62` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:55.506801 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36440c36-8268-4a56-b402-3a4438e431c0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=45dd29a3dd7645699789116580436b62)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:55.508711 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36440c36-8268-4a56-b402-3a4438e431c0 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:55.510260 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:50:55.645203 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-36440c36-8268-4a56-b402-3a4438e431c0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '45dd29a3dd7645699789116580436b62', 'request_id': 'req-36440c36-8268-4a56-b402-3a4438e431c0'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:55.650870 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-36440c36-8268-4a56-b402-3a4438e431c0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '45dd29a3dd7645699789116580436b62', 'request_id': 'req-36440c36-8268-4a56-b402-3a4438e431c0'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:55.747815 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-36440c36-8268-4a56-b402-3a4438e431c0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '45dd29a3dd7645699789116580436b62', 'request_id': 'req-36440c36-8268-4a56-b402-3a4438e431c0'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:55.760427 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-36440c36-8268-4a56-b402-3a4438e431c0 admin admin] The token cache is being invalidate because project 45dd29a3dd7645699789116580436b62 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:50:55.761637 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-36440c36-8268-4a56-b402-3a4438e431c0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:55.762507 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-36440c36-8268-4a56-b402-3a4438e431c0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:50:55.764367 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1425/2820] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:50:55 2022] DELETE /identity/v3/projects/45dd29a3dd7645699789116580436b62 => generated 0 bytes in 340 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:50:55.769432 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3b6f56-733e-43b7-983d-f84151381a37 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:55.774387 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3b6f56-733e-43b7-983d-f84151381a37 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:55.774387 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3b6f56-733e-43b7-983d-f84151381a37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:55.790117 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-1b3b6f56-733e-43b7-983d-f84151381a37 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:50:55.791389 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-1b3b6f56-733e-43b7-983d-f84151381a37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:55.853109 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1396/2821] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:50:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:50:55.860994 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bddb40-6b68-419a-801f-5a401078f7cc None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:55.861319 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bddb40-6b68-419a-801f-5a401078f7cc None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:55.861606 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bddb40-6b68-419a-801f-5a401078f7cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:55.879943 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-10bddb40-6b68-419a-801f-5a401078f7cc None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:50:55.881205 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-10bddb40-6b68-419a-801f-5a401078f7cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:55.889331 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1426/2822] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:50:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:50:55.893557 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3b6f56-733e-43b7-983d-f84151381a37 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:55.904125 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3b6f56-733e-43b7-983d-f84151381a37 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:55.908844 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5815c81-daf3-43cc-bd0d-0e1bb4ea2aea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:55.909504 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5815c81-daf3-43cc-bd0d-0e1bb4ea2aea admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:55.909797 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5815c81-daf3-43cc-bd0d-0e1bb4ea2aea admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:55.910220 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5815c81-daf3-43cc-bd0d-0e1bb4ea2aea admin admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:55.910989 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5815c81-daf3-43cc-bd0d-0e1bb4ea2aea admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:50:55.911302 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5815c81-daf3-43cc-bd0d-0e1bb4ea2aea admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:55.913891 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5815c81-daf3-43cc-bd0d-0e1bb4ea2aea admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:55.926227 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1397/2823] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:50:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:50:55.931250 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10bddb40-6b68-419a-801f-5a401078f7cc None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:55.941413 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10bddb40-6b68-419a-801f-5a401078f7cc None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:55.946290 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24c771f-86d1-45bd-bf29-be1f9b1ea81e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:55.947072 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24c771f-86d1-45bd-bf29-be1f9b1ea81e admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:55.947344 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24c771f-86d1-45bd-bf29-be1f9b1ea81e admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:55.947621 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24c771f-86d1-45bd-bf29-be1f9b1ea81e admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:55.948314 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24c771f-86d1-45bd-bf29-be1f9b1ea81e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:55.950211 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24c771f-86d1-45bd-bf29-be1f9b1ea81e admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:56.063686 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1427/2824] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:50:55 2022] POST /identity/v3/projects => generated 385 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:50:56.068257 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5815c81-daf3-43cc-bd0d-0e1bb4ea2aea admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:56.078476 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5815c81-daf3-43cc-bd0d-0e1bb4ea2aea admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:56.138303 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6203b34-c78f-4da3-949b-19c56a82e3c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:56.139007 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6203b34-c78f-4da3-949b-19c56a82e3c8 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:56.139296 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6203b34-c78f-4da3-949b-19c56a82e3c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:56.139577 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6203b34-c78f-4da3-949b-19c56a82e3c8 admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:56.140263 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6203b34-c78f-4da3-949b-19c56a82e3c8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:56.142174 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6203b34-c78f-4da3-949b-19c56a82e3c8 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:56.152507 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:50:56.217425 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1398/2825] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:50:56 2022] POST /identity/v3/users => generated 293 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:50:56.221749 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24c771f-86d1-45bd-bf29-be1f9b1ea81e admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:56.232743 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24c771f-86d1-45bd-bf29-be1f9b1ea81e admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:56.237613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8793de0e-75dd-4afc-abe0-08c5c0a20d2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:56.238310 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8793de0e-75dd-4afc-abe0-08c5c0a20d2e admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:56.238633 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8793de0e-75dd-4afc-abe0-08c5c0a20d2e admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:56.238921 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8793de0e-75dd-4afc-abe0-08c5c0a20d2e admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:56.239669 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8793de0e-75dd-4afc-abe0-08c5c0a20d2e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:56.241388 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8793de0e-75dd-4afc-abe0-08c5c0a20d2e admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:56.252222 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1428/2826] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:50:56 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:50:56.257142 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6203b34-c78f-4da3-949b-19c56a82e3c8 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:56.267808 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6203b34-c78f-4da3-949b-19c56a82e3c8 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:56.273452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecb0d65-7369-4bba-8072-04c5a35efa8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:56.274120 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecb0d65-7369-4bba-8072-04c5a35efa8a admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:56.274412 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecb0d65-7369-4bba-8072-04c5a35efa8a admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:56.274736 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecb0d65-7369-4bba-8072-04c5a35efa8a admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:56.275475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecb0d65-7369-4bba-8072-04c5a35efa8a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:56.277094 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecb0d65-7369-4bba-8072-04c5a35efa8a admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:56.288780 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1399/2827] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:50:56 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:50:56.293193 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8793de0e-75dd-4afc-abe0-08c5c0a20d2e admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:56.303610 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8793de0e-75dd-4afc-abe0-08c5c0a20d2e admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:56.308595 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2afce773-cf37-4d17-aa5a-dba180e0f5fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:56.309479 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afce773-cf37-4d17-aa5a-dba180e0f5fe admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:56.309768 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afce773-cf37-4d17-aa5a-dba180e0f5fe admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:56.310051 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afce773-cf37-4d17-aa5a-dba180e0f5fe admin admin] PATH_INFO: `/v3/projects/1a1f5a3f981b4361999abf1f8a1045cd/users/6f9a1a5397cd4a3b8949b74c1bfd2ad5/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:56.326900 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2afce773-cf37-4d17-aa5a-dba180e0f5fe admin admin] RBAC: Authorizing `identity:create_grant(project_id=1a1f5a3f981b4361999abf1f8a1045cd, user_id=6f9a1a5397cd4a3b8949b74c1bfd2ad5, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:56.329663 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2afce773-cf37-4d17-aa5a-dba180e0f5fe admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:56.406171 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1429/2828] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:50:56 2022] PUT /identity/v3/projects/1a1f5a3f981b4361999abf1f8a1045cd/users/6f9a1a5397cd4a3b8949b74c1bfd2ad5/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:50:56.417952 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1c5132-44be-4d42-967e-8e8f25c5b552 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:56.418296 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1c5132-44be-4d42-967e-8e8f25c5b552 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:56.418694 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1c5132-44be-4d42-967e-8e8f25c5b552 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:56.445882 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-ed1c5132-44be-4d42-967e-8e8f25c5b552 None None] MFA Rules not processed for user `6f9a1a5397cd4a3b8949b74c1bfd2ad5`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:50:56.447168 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-ed1c5132-44be-4d42-967e-8e8f25c5b552 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:56.516036 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1400/2829] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:50:56 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:50:56.527491 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2afce773-cf37-4d17-aa5a-dba180e0f5fe admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:50:56.529273 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-2afce773-cf37-4d17-aa5a-dba180e0f5fe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:50:56.591131 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2afce773-cf37-4d17-aa5a-dba180e0f5fe admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:50:56.594947 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ea1b90-bfab-4fb4-8a6b-f41ad0dea612 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:50:56.595645 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ea1b90-bfab-4fb4-8a6b-f41ad0dea612 service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:50:56.595934 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ea1b90-bfab-4fb4-8a6b-f41ad0dea612 service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:50:56.596217 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ea1b90-bfab-4fb4-8a6b-f41ad0dea612 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:50:56.605165 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36ea1b90-bfab-4fb4-8a6b-f41ad0dea612 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:50:56.606658 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36ea1b90-bfab-4fb4-8a6b-f41ad0dea612 service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:50:56.616321 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1430/2830] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:50:56 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:51:04.887581 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1c5132-44be-4d42-967e-8e8f25c5b552 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:04.889643 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-ed1c5132-44be-4d42-967e-8e8f25c5b552 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:51:04.961754 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1c5132-44be-4d42-967e-8e8f25c5b552 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:04.966884 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad260525-6cab-4d61-969e-bffaa69e320f service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f623be28c83247c78d3ea6873c347fe0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:04.968228 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad260525-6cab-4d61-969e-bffaa69e320f service placement] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:04.968228 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad260525-6cab-4d61-969e-bffaa69e320f service placement] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:04.968228 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad260525-6cab-4d61-969e-bffaa69e320f service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:04.969944 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-ad260525-6cab-4d61-969e-bffaa69e320f service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:51:04.982127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad260525-6cab-4d61-969e-bffaa69e320f service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:04.983788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad260525-6cab-4d61-969e-bffaa69e320f service placement] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:04.994832 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1401/2831] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:51:04 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:51:05.031358 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ea1b90-bfab-4fb4-8a6b-f41ad0dea612 service cinder] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:05.031358 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-36ea1b90-bfab-4fb4-8a6b-f41ad0dea612 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:51:05.063585 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:51:05.098722 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ea1b90-bfab-4fb4-8a6b-f41ad0dea612 service cinder] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:05.104482 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1d8550-2dfb-4aa5-a66a-b4529a737bde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:05.105370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1d8550-2dfb-4aa5-a66a-b4529a737bde admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:05.105654 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1d8550-2dfb-4aa5-a66a-b4529a737bde admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:05.105994 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1d8550-2dfb-4aa5-a66a-b4529a737bde admin admin] PATH_INFO: `/v3/users/01d3c959a87247cd833db555fdef9af5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:05.107230 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1d8550-2dfb-4aa5-a66a-b4529a737bde admin admin] RBAC: Authorizing `identity:delete_user(user_id=01d3c959a87247cd833db555fdef9af5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:05.109296 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1d8550-2dfb-4aa5-a66a-b4529a737bde admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:05.316393 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-6e1d8550-2dfb-4aa5-a66a-b4529a737bde admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '01d3c959a87247cd833db555fdef9af5'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:05.317071 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-6e1d8550-2dfb-4aa5-a66a-b4529a737bde admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '01d3c959a87247cd833db555fdef9af5'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:05.317698 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-6e1d8550-2dfb-4aa5-a66a-b4529a737bde admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '01d3c959a87247cd833db555fdef9af5'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:05.324572 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-6e1d8550-2dfb-4aa5-a66a-b4529a737bde admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '01d3c959a87247cd833db555fdef9af5'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:05.337070 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-6e1d8550-2dfb-4aa5-a66a-b4529a737bde admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '01d3c959a87247cd833db555fdef9af5'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:05.411004 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1431/2832] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:51:05 2022] DELETE /identity/v3/users/01d3c959a87247cd833db555fdef9af5 => generated 0 bytes in 383 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:51:05.487382 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad260525-6cab-4d61-969e-bffaa69e320f service placement] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:05.499163 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad260525-6cab-4d61-969e-bffaa69e320f service placement] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:05.562663 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47adee12-7be3-470a-940f-7d608762a430 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:05.563829 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47adee12-7be3-470a-940f-7d608762a430 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:05.564283 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47adee12-7be3-470a-940f-7d608762a430 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:05.565002 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47adee12-7be3-470a-940f-7d608762a430 admin admin] PATH_INFO: `/v3/users/6f9a1a5397cd4a3b8949b74c1bfd2ad5` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:05.566534 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47adee12-7be3-470a-940f-7d608762a430 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6f9a1a5397cd4a3b8949b74c1bfd2ad5)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:05.568887 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47adee12-7be3-470a-940f-7d608762a430 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:05.577962 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=100 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:51:05.698763 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1d8550-2dfb-4aa5-a66a-b4529a737bde admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:05.700331 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-6e1d8550-2dfb-4aa5-a66a-b4529a737bde admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:51:05.714762 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1d8550-2dfb-4aa5-a66a-b4529a737bde admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:05.719757 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b028c7b-80fb-441b-88f7-b8806ea3fdf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:05.721664 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b028c7b-80fb-441b-88f7-b8806ea3fdf2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:05.721664 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b028c7b-80fb-441b-88f7-b8806ea3fdf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:05.721664 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b028c7b-80fb-441b-88f7-b8806ea3fdf2 admin admin] PATH_INFO: `/v3/projects/c3b0319262bc468d82fa3fba94540142` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:05.722520 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b028c7b-80fb-441b-88f7-b8806ea3fdf2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c3b0319262bc468d82fa3fba94540142)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:05.724454 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b028c7b-80fb-441b-88f7-b8806ea3fdf2 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:05.771303 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-47adee12-7be3-470a-940f-7d608762a430 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6f9a1a5397cd4a3b8949b74c1bfd2ad5'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:05.818056 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-0b028c7b-80fb-441b-88f7-b8806ea3fdf2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c3b0319262bc468d82fa3fba94540142', 'request_id': 'req-0b028c7b-80fb-441b-88f7-b8806ea3fdf2'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:05.823887 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-0b028c7b-80fb-441b-88f7-b8806ea3fdf2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c3b0319262bc468d82fa3fba94540142', 'request_id': 'req-0b028c7b-80fb-441b-88f7-b8806ea3fdf2'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:05.855711 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-47adee12-7be3-470a-940f-7d608762a430 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6f9a1a5397cd4a3b8949b74c1bfd2ad5'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:05.856330 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-47adee12-7be3-470a-940f-7d608762a430 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6f9a1a5397cd4a3b8949b74c1bfd2ad5'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:05.863833 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-47adee12-7be3-470a-940f-7d608762a430 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6f9a1a5397cd4a3b8949b74c1bfd2ad5'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:05.876857 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-47adee12-7be3-470a-940f-7d608762a430 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6f9a1a5397cd4a3b8949b74c1bfd2ad5'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:05.882914 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1402/2833] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:51:05 2022] DELETE /identity/v3/users/6f9a1a5397cd4a3b8949b74c1bfd2ad5 => generated 0 bytes in 396 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:51:05.887274 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1767545d-f77a-480e-8ee9-ed1b1f4940e6 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:05.887569 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1767545d-f77a-480e-8ee9-ed1b1f4940e6 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:05.887853 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1767545d-f77a-480e-8ee9-ed1b1f4940e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:05.905265 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-1767545d-f77a-480e-8ee9-ed1b1f4940e6 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:51:05.906488 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-1767545d-f77a-480e-8ee9-ed1b1f4940e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:51:05.924537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-0b028c7b-80fb-441b-88f7-b8806ea3fdf2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c3b0319262bc468d82fa3fba94540142', 'request_id': 'req-0b028c7b-80fb-441b-88f7-b8806ea3fdf2'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:05.937608 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-0b028c7b-80fb-441b-88f7-b8806ea3fdf2 admin admin] The token cache is being invalidate because project c3b0319262bc468d82fa3fba94540142 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:51:05.937817 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-0b028c7b-80fb-441b-88f7-b8806ea3fdf2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:05.938528 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-0b028c7b-80fb-441b-88f7-b8806ea3fdf2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:05.940409 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1432/2834] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:51:05 2022] DELETE /identity/v3/projects/c3b0319262bc468d82fa3fba94540142 => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:51:05.971332 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1403/2835] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:51:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:51:05.982114 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b028c7b-80fb-441b-88f7-b8806ea3fdf2 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:05.983670 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-0b028c7b-80fb-441b-88f7-b8806ea3fdf2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:51:06.024181 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:51:06.052959 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b028c7b-80fb-441b-88f7-b8806ea3fdf2 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:06.110281 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1481a9-bd2d-4ef8-8e4d-2c5e9f0c74c8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:06.110983 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1481a9-bd2d-4ef8-8e4d-2c5e9f0c74c8 service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:06.111275 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1481a9-bd2d-4ef8-8e4d-2c5e9f0c74c8 service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:06.111556 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1481a9-bd2d-4ef8-8e4d-2c5e9f0c74c8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:06.123360 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1481a9-bd2d-4ef8-8e4d-2c5e9f0c74c8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:06.124863 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1481a9-bd2d-4ef8-8e4d-2c5e9f0c74c8 service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:06.136105 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1433/2836] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:51:05 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:51:06.590159 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1767545d-f77a-480e-8ee9-ed1b1f4940e6 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:06.591641 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-1767545d-f77a-480e-8ee9-ed1b1f4940e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:51:06.604305 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1767545d-f77a-480e-8ee9-ed1b1f4940e6 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:06.609050 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-615d8921-707a-4779-ba41-8c43a01156e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:06.609909 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615d8921-707a-4779-ba41-8c43a01156e0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:06.610200 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615d8921-707a-4779-ba41-8c43a01156e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:06.611917 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615d8921-707a-4779-ba41-8c43a01156e0 admin admin] PATH_INFO: `/v3/projects/1a1f5a3f981b4361999abf1f8a1045cd` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:06.611917 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-615d8921-707a-4779-ba41-8c43a01156e0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1a1f5a3f981b4361999abf1f8a1045cd)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:06.613628 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-615d8921-707a-4779-ba41-8c43a01156e0 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:06.746115 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-615d8921-707a-4779-ba41-8c43a01156e0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1a1f5a3f981b4361999abf1f8a1045cd', 'request_id': 'req-615d8921-707a-4779-ba41-8c43a01156e0'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:06.752484 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-615d8921-707a-4779-ba41-8c43a01156e0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1a1f5a3f981b4361999abf1f8a1045cd', 'request_id': 'req-615d8921-707a-4779-ba41-8c43a01156e0'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:06.758720 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-615d8921-707a-4779-ba41-8c43a01156e0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1a1f5a3f981b4361999abf1f8a1045cd', 'request_id': 'req-615d8921-707a-4779-ba41-8c43a01156e0'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:06.853127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-615d8921-707a-4779-ba41-8c43a01156e0 admin admin] The token cache is being invalidate because project 1a1f5a3f981b4361999abf1f8a1045cd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:51:06.853323 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-615d8921-707a-4779-ba41-8c43a01156e0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:06.853954 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-615d8921-707a-4779-ba41-8c43a01156e0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:06.855564 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1404/2837] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:51:06 2022] DELETE /identity/v3/projects/1a1f5a3f981b4361999abf1f8a1045cd => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:51:06.860570 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fcba3d-ec95-45ea-b2cd-abcfed7081c9 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:06.860861 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fcba3d-ec95-45ea-b2cd-abcfed7081c9 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:06.861144 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8fcba3d-ec95-45ea-b2cd-abcfed7081c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:06.879867 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-d8fcba3d-ec95-45ea-b2cd-abcfed7081c9 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:51:06.880759 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-d8fcba3d-ec95-45ea-b2cd-abcfed7081c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:51:06.949158 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1434/2838] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:51:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:51:06.956965 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e6fb5e-fa48-4b4f-a73f-44ba35d2da14 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:06.957487 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e6fb5e-fa48-4b4f-a73f-44ba35d2da14 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:06.957614 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e6fb5e-fa48-4b4f-a73f-44ba35d2da14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:06.970288 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:51:06.976451 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-11e6fb5e-fa48-4b4f-a73f-44ba35d2da14 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:51:06.977923 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-11e6fb5e-fa48-4b4f-a73f-44ba35d2da14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:51:06.986066 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1405/2839] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:51:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:51:06.990353 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fcba3d-ec95-45ea-b2cd-abcfed7081c9 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:07.001047 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8fcba3d-ec95-45ea-b2cd-abcfed7081c9 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:07.005872 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac3efda-81c4-459b-9b64-f7e962c679eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:07.006704 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac3efda-81c4-459b-9b64-f7e962c679eb admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:07.007002 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac3efda-81c4-459b-9b64-f7e962c679eb admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:07.007287 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac3efda-81c4-459b-9b64-f7e962c679eb admin admin] PATH_INFO: `/v3/domains` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:07.008633 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac3efda-81c4-459b-9b64-f7e962c679eb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80452) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:51:07.008994 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac3efda-81c4-459b-9b64-f7e962c679eb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:07.010672 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac3efda-81c4-459b-9b64-f7e962c679eb admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:07.023984 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1435/2840] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:51:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 12 17:51:07.028386 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e6fb5e-fa48-4b4f-a73f-44ba35d2da14 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:07.037941 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e6fb5e-fa48-4b4f-a73f-44ba35d2da14 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:07.043403 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593e8879-43de-4e6b-8103-c06674d7f8db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:07.044077 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593e8879-43de-4e6b-8103-c06674d7f8db admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:07.044369 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593e8879-43de-4e6b-8103-c06674d7f8db admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:07.044648 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593e8879-43de-4e6b-8103-c06674d7f8db admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:07.045351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-593e8879-43de-4e6b-8103-c06674d7f8db admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:07.047373 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-593e8879-43de-4e6b-8103-c06674d7f8db admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:07.153112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1406/2841] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:51:07 2022] POST /identity/v3/projects => generated 373 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:51:07.157226 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac3efda-81c4-459b-9b64-f7e962c679eb admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:07.167669 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac3efda-81c4-459b-9b64-f7e962c679eb admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:07.226063 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea45d4e-aee4-4a7c-a8d4-c6ba53465445 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:07.226843 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea45d4e-aee4-4a7c-a8d4-c6ba53465445 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:07.227241 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea45d4e-aee4-4a7c-a8d4-c6ba53465445 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:07.227709 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea45d4e-aee4-4a7c-a8d4-c6ba53465445 admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:07.228417 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea45d4e-aee4-4a7c-a8d4-c6ba53465445 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:07.230356 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea45d4e-aee4-4a7c-a8d4-c6ba53465445 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:07.325157 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1436/2842] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:51:07 2022] POST /identity/v3/users => generated 287 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:51:07.329263 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593e8879-43de-4e6b-8103-c06674d7f8db admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:07.339713 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593e8879-43de-4e6b-8103-c06674d7f8db admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:07.344486 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a532e3-3b9a-4de3-b9cc-19d87c2ac2b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:07.345144 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a532e3-3b9a-4de3-b9cc-19d87c2ac2b1 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:07.345431 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a532e3-3b9a-4de3-b9cc-19d87c2ac2b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:07.345712 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a532e3-3b9a-4de3-b9cc-19d87c2ac2b1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:07.346448 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a532e3-3b9a-4de3-b9cc-19d87c2ac2b1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:07.348048 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80a532e3-3b9a-4de3-b9cc-19d87c2ac2b1 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:07.359127 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1407/2843] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:51:07 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 12 17:51:07.363005 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea45d4e-aee4-4a7c-a8d4-c6ba53465445 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:07.373681 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea45d4e-aee4-4a7c-a8d4-c6ba53465445 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:07.378507 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85652250-8a45-486a-88ea-d519d1c92437 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:07.379154 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85652250-8a45-486a-88ea-d519d1c92437 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:07.379441 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85652250-8a45-486a-88ea-d519d1c92437 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:07.379720 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85652250-8a45-486a-88ea-d519d1c92437 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:07.380448 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85652250-8a45-486a-88ea-d519d1c92437 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:07.382036 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85652250-8a45-486a-88ea-d519d1c92437 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:07.383558 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:51:07.393368 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1437/2844] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:51:07 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:51:07.402137 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a532e3-3b9a-4de3-b9cc-19d87c2ac2b1 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:07.412448 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80a532e3-3b9a-4de3-b9cc-19d87c2ac2b1 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:07.417859 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212a6fa9-864e-426e-a71f-21a510f4440e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:07.418835 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212a6fa9-864e-426e-a71f-21a510f4440e admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:07.419126 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212a6fa9-864e-426e-a71f-21a510f4440e admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:07.419440 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212a6fa9-864e-426e-a71f-21a510f4440e admin admin] PATH_INFO: `/v3/projects/28ecabca5fb6412ca7028252ef0ab5dd/users/b11ba323498a440cbc8e234228e9f995/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:07.434903 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-212a6fa9-864e-426e-a71f-21a510f4440e admin admin] RBAC: Authorizing `identity:create_grant(project_id=28ecabca5fb6412ca7028252ef0ab5dd, user_id=b11ba323498a440cbc8e234228e9f995, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:07.437857 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-212a6fa9-864e-426e-a71f-21a510f4440e admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:07.502086 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1408/2845] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:51:07 2022] PUT /identity/v3/projects/28ecabca5fb6412ca7028252ef0ab5dd/users/b11ba323498a440cbc8e234228e9f995/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:51:07.512977 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c7bf74-15eb-475f-812a-464db36f23a3 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:07.513273 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c7bf74-15eb-475f-812a-464db36f23a3 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:07.513553 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c7bf74-15eb-475f-812a-464db36f23a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:07.541329 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-48c7bf74-15eb-475f-812a-464db36f23a3 None None] MFA Rules not processed for user `b11ba323498a440cbc8e234228e9f995`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:51:07.542604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-48c7bf74-15eb-475f-812a-464db36f23a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:51:07.608471 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1438/2846] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:51:07 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:51:07.613156 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212a6fa9-864e-426e-a71f-21a510f4440e admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:07.676084 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212a6fa9-864e-426e-a71f-21a510f4440e admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:07.681674 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fed9d7a-baff-4771-bd27-b401ff4f9bd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:07.682349 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fed9d7a-baff-4771-bd27-b401ff4f9bd6 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:07.682709 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fed9d7a-baff-4771-bd27-b401ff4f9bd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:07.684357 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fed9d7a-baff-4771-bd27-b401ff4f9bd6 admin admin] PATH_INFO: `/v3/projects` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:07.684357 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fed9d7a-baff-4771-bd27-b401ff4f9bd6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:07.685615 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fed9d7a-baff-4771-bd27-b401ff4f9bd6 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:07.783749 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1409/2847] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:51:07 2022] POST /identity/v3/projects => generated 371 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:51:07.788158 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c7bf74-15eb-475f-812a-464db36f23a3 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:07.799811 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c7bf74-15eb-475f-812a-464db36f23a3 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:07.868683 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0dae86-89be-4520-acc6-4738c6997120 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:07.869827 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0dae86-89be-4520-acc6-4738c6997120 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:07.870347 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0dae86-89be-4520-acc6-4738c6997120 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:07.870877 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0dae86-89be-4520-acc6-4738c6997120 admin admin] PATH_INFO: `/v3/users` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:07.872176 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b0dae86-89be-4520-acc6-4738c6997120 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:07.875892 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b0dae86-89be-4520-acc6-4738c6997120 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:08.024126 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1439/2848] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:51:07 2022] POST /identity/v3/users => generated 292 bytes in 236 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:51:08.028130 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fed9d7a-baff-4771-bd27-b401ff4f9bd6 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:08.038565 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fed9d7a-baff-4771-bd27-b401ff4f9bd6 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:08.043985 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ece73ce-241e-4807-94da-3cbfa3ad3c2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:08.044666 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ece73ce-241e-4807-94da-3cbfa3ad3c2c admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:08.044993 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ece73ce-241e-4807-94da-3cbfa3ad3c2c admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:08.045281 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ece73ce-241e-4807-94da-3cbfa3ad3c2c admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:08.046019 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ece73ce-241e-4807-94da-3cbfa3ad3c2c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:08.047773 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ece73ce-241e-4807-94da-3cbfa3ad3c2c admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:08.058961 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1410/2849] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:51:08 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:51:08.063357 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0dae86-89be-4520-acc6-4738c6997120 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:08.074764 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0dae86-89be-4520-acc6-4738c6997120 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:08.079515 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aea520e-5f02-4527-bf6c-0cb0262a952b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:08.080414 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aea520e-5f02-4527-bf6c-0cb0262a952b admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:08.080704 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aea520e-5f02-4527-bf6c-0cb0262a952b admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:08.080992 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aea520e-5f02-4527-bf6c-0cb0262a952b admin admin] PATH_INFO: `/v3/projects/971cc9851f914a9db11a69d9285a5ffb/users/89cb0990d9bb459f959fd49b0406ae25/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:08.099066 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aea520e-5f02-4527-bf6c-0cb0262a952b admin admin] RBAC: Authorizing `identity:create_grant(project_id=971cc9851f914a9db11a69d9285a5ffb, user_id=89cb0990d9bb459f959fd49b0406ae25, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:08.101889 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aea520e-5f02-4527-bf6c-0cb0262a952b admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:08.203208 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1440/2850] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:51:08 2022] PUT /identity/v3/projects/971cc9851f914a9db11a69d9285a5ffb/users/89cb0990d9bb459f959fd49b0406ae25/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:51:08.207408 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ece73ce-241e-4807-94da-3cbfa3ad3c2c admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:08.218610 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ece73ce-241e-4807-94da-3cbfa3ad3c2c admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:08.276500 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144d9a80-ca66-4c4a-a385-20c941717dd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:08.277180 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144d9a80-ca66-4c4a-a385-20c941717dd2 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:08.277472 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144d9a80-ca66-4c4a-a385-20c941717dd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:08.277756 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144d9a80-ca66-4c4a-a385-20c941717dd2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:08.278601 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144d9a80-ca66-4c4a-a385-20c941717dd2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:08.280242 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144d9a80-ca66-4c4a-a385-20c941717dd2 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:08.290659 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1411/2851] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:51:08 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:51:08.295129 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aea520e-5f02-4527-bf6c-0cb0262a952b admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:08.305917 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aea520e-5f02-4527-bf6c-0cb0262a952b admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:08.310700 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7619509-5f7b-4b46-8755-332cbf9086f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:08.311581 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7619509-5f7b-4b46-8755-332cbf9086f3 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:08.311866 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7619509-5f7b-4b46-8755-332cbf9086f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:08.312145 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7619509-5f7b-4b46-8755-332cbf9086f3 admin admin] PATH_INFO: `/v3/projects/971cc9851f914a9db11a69d9285a5ffb/users/89cb0990d9bb459f959fd49b0406ae25/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:08.314135 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7619509-5f7b-4b46-8755-332cbf9086f3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=971cc9851f914a9db11a69d9285a5ffb, user_id=89cb0990d9bb459f959fd49b0406ae25, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:08.317043 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7619509-5f7b-4b46-8755-332cbf9086f3 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:08.339064 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1441/2852] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:51:08 2022] PUT /identity/v3/projects/971cc9851f914a9db11a69d9285a5ffb/users/89cb0990d9bb459f959fd49b0406ae25/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:51:08.350523 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a962f8-9920-42a4-8e7d-1e26a6e6a94a None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:08.350901 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a962f8-9920-42a4-8e7d-1e26a6e6a94a None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:08.351254 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a962f8-9920-42a4-8e7d-1e26a6e6a94a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:08.375504 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:51:08.378419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-a3a962f8-9920-42a4-8e7d-1e26a6e6a94a None None] MFA Rules not processed for user `89cb0990d9bb459f959fd49b0406ae25`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:51:08.379684 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-a3a962f8-9920-42a4-8e7d-1e26a6e6a94a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:51:08.449520 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1412/2853] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:51:08 2022] POST /identity/v3/auth/tokens => generated 3305 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:51:08.464244 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7619509-5f7b-4b46-8755-332cbf9086f3 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:08.465981 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-b7619509-5f7b-4b46-8755-332cbf9086f3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:51:08.510941 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:51:08.535242 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7619509-5f7b-4b46-8755-332cbf9086f3 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:08.535242 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de4ae32-996a-4249-8c2d-26b1b46d23cb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f6a82b49464846d1a207189f17fd4cf7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:08.535242 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de4ae32-996a-4249-8c2d-26b1b46d23cb service cinder] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:08.535242 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de4ae32-996a-4249-8c2d-26b1b46d23cb service cinder] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:08.535902 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de4ae32-996a-4249-8c2d-26b1b46d23cb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:08.593514 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5de4ae32-996a-4249-8c2d-26b1b46d23cb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:08.595011 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5de4ae32-996a-4249-8c2d-26b1b46d23cb service cinder] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:08.604367 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1442/2854] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:51:08 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:51:11.693475 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a962f8-9920-42a4-8e7d-1e26a6e6a94a None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:11.755237 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a962f8-9920-42a4-8e7d-1e26a6e6a94a None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:11.760378 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8645831-237c-4130-b7b4-b45629b9472d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:11.761258 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8645831-237c-4130-b7b4-b45629b9472d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:11.761548 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8645831-237c-4130-b7b4-b45629b9472d admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:11.761830 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8645831-237c-4130-b7b4-b45629b9472d admin admin] PATH_INFO: `/v3/users/b11ba323498a440cbc8e234228e9f995` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:11.763076 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8645831-237c-4130-b7b4-b45629b9472d admin admin] RBAC: Authorizing `identity:delete_user(user_id=b11ba323498a440cbc8e234228e9f995)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:11.765019 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8645831-237c-4130-b7b4-b45629b9472d admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:11.938612 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-e8645831-237c-4130-b7b4-b45629b9472d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b11ba323498a440cbc8e234228e9f995'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:12.018891 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-e8645831-237c-4130-b7b4-b45629b9472d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b11ba323498a440cbc8e234228e9f995'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:12.019616 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-e8645831-237c-4130-b7b4-b45629b9472d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b11ba323498a440cbc8e234228e9f995'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:12.027125 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-e8645831-237c-4130-b7b4-b45629b9472d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b11ba323498a440cbc8e234228e9f995'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:12.039483 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-e8645831-237c-4130-b7b4-b45629b9472d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b11ba323498a440cbc8e234228e9f995'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:12.045671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1413/2855] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:51:11 2022] DELETE /identity/v3/users/b11ba323498a440cbc8e234228e9f995 => generated 0 bytes in 353 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:51:12.050335 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b13a300-b40c-4318-bf6c-628271c95f3c None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:12.050874 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b13a300-b40c-4318-bf6c-628271c95f3c None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:12.050973 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b13a300-b40c-4318-bf6c-628271c95f3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:12.068668 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-1b13a300-b40c-4318-bf6c-628271c95f3c None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:51:12.069861 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-1b13a300-b40c-4318-bf6c-628271c95f3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:51:12.128366 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1443/2856] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:51:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:51:12.138143 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8645831-237c-4130-b7b4-b45629b9472d admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:12.139633 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-e8645831-237c-4130-b7b4-b45629b9472d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:51:12.159803 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:51:12.200393 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8645831-237c-4130-b7b4-b45629b9472d admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:12.204334 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3e58a8-eac2-49b8-8417-33738f921670 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:12.204998 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3e58a8-eac2-49b8-8417-33738f921670 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:12.205293 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3e58a8-eac2-49b8-8417-33738f921670 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:12.205579 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3e58a8-eac2-49b8-8417-33738f921670 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:12.217094 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b3e58a8-eac2-49b8-8417-33738f921670 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:12.220160 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b3e58a8-eac2-49b8-8417-33738f921670 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:12.229452 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1414/2857] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:51:12 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:51:12.639081 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b13a300-b40c-4318-bf6c-628271c95f3c None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:12.640616 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-1b13a300-b40c-4318-bf6c-628271c95f3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:51:12.653810 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b13a300-b40c-4318-bf6c-628271c95f3c None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:12.659023 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4fc50c0-a1c7-4cc0-ace5-3bce0076742f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:12.659979 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4fc50c0-a1c7-4cc0-ace5-3bce0076742f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:12.660289 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4fc50c0-a1c7-4cc0-ace5-3bce0076742f admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:12.660574 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4fc50c0-a1c7-4cc0-ace5-3bce0076742f admin admin] PATH_INFO: `/v3/projects/28ecabca5fb6412ca7028252ef0ab5dd` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:12.661778 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4fc50c0-a1c7-4cc0-ace5-3bce0076742f admin admin] RBAC: Authorizing `identity:delete_project(project_id=28ecabca5fb6412ca7028252ef0ab5dd)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:12.663848 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4fc50c0-a1c7-4cc0-ace5-3bce0076742f admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:12.782502 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-f4fc50c0-a1c7-4cc0-ace5-3bce0076742f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '28ecabca5fb6412ca7028252ef0ab5dd', 'request_id': 'req-f4fc50c0-a1c7-4cc0-ace5-3bce0076742f'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:12.788067 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-f4fc50c0-a1c7-4cc0-ace5-3bce0076742f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '28ecabca5fb6412ca7028252ef0ab5dd', 'request_id': 'req-f4fc50c0-a1c7-4cc0-ace5-3bce0076742f'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:12.855778 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-f4fc50c0-a1c7-4cc0-ace5-3bce0076742f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '28ecabca5fb6412ca7028252ef0ab5dd', 'request_id': 'req-f4fc50c0-a1c7-4cc0-ace5-3bce0076742f'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:12.868400 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-f4fc50c0-a1c7-4cc0-ace5-3bce0076742f admin admin] The token cache is being invalidate because project 28ecabca5fb6412ca7028252ef0ab5dd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:51:12.868673 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-f4fc50c0-a1c7-4cc0-ace5-3bce0076742f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:12.869331 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-f4fc50c0-a1c7-4cc0-ace5-3bce0076742f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:12.870969 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1444/2858] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:51:12 2022] DELETE /identity/v3/projects/28ecabca5fb6412ca7028252ef0ab5dd => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:51:12.875014 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3e58a8-eac2-49b8-8417-33738f921670 service neutron] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:12.885674 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3e58a8-eac2-49b8-8417-33738f921670 service neutron] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:12.945834 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035ad969-63ec-477c-aaab-3b52b7ee0be6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:12.946754 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035ad969-63ec-477c-aaab-3b52b7ee0be6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:12.947047 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035ad969-63ec-477c-aaab-3b52b7ee0be6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:12.950939 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035ad969-63ec-477c-aaab-3b52b7ee0be6 admin admin] PATH_INFO: `/v3/users/89cb0990d9bb459f959fd49b0406ae25` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:12.950939 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035ad969-63ec-477c-aaab-3b52b7ee0be6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=89cb0990d9bb459f959fd49b0406ae25)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:12.950939 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035ad969-63ec-477c-aaab-3b52b7ee0be6 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:13.133372 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-035ad969-63ec-477c-aaab-3b52b7ee0be6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '89cb0990d9bb459f959fd49b0406ae25'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:13.216082 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-035ad969-63ec-477c-aaab-3b52b7ee0be6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '89cb0990d9bb459f959fd49b0406ae25'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:13.216689 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-035ad969-63ec-477c-aaab-3b52b7ee0be6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '89cb0990d9bb459f959fd49b0406ae25'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:13.224350 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-035ad969-63ec-477c-aaab-3b52b7ee0be6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '89cb0990d9bb459f959fd49b0406ae25'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:13.247252 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-035ad969-63ec-477c-aaab-3b52b7ee0be6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '89cb0990d9bb459f959fd49b0406ae25'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:13.254463 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1415/2859] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:51:12 2022] DELETE /identity/v3/users/89cb0990d9bb459f959fd49b0406ae25 => generated 0 bytes in 380 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:51:13.720248 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4fc50c0-a1c7-4cc0-ace5-3bce0076742f admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:13.720248 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-f4fc50c0-a1c7-4cc0-ace5-3bce0076742f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:51:13.733782 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4fc50c0-a1c7-4cc0-ace5-3bce0076742f admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:13.754720 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:51:13.794897 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7156c9c9-15ed-4abc-bc33-8ca60866b1cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:13.795764 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7156c9c9-15ed-4abc-bc33-8ca60866b1cc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:13.796047 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7156c9c9-15ed-4abc-bc33-8ca60866b1cc admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:13.796326 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7156c9c9-15ed-4abc-bc33-8ca60866b1cc admin admin] PATH_INFO: `/v3/projects/971cc9851f914a9db11a69d9285a5ffb` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:13.797553 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7156c9c9-15ed-4abc-bc33-8ca60866b1cc admin admin] RBAC: Authorizing `identity:delete_project(project_id=971cc9851f914a9db11a69d9285a5ffb)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:13.799501 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7156c9c9-15ed-4abc-bc33-8ca60866b1cc admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:13.922537 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-7156c9c9-15ed-4abc-bc33-8ca60866b1cc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '971cc9851f914a9db11a69d9285a5ffb', 'request_id': 'req-7156c9c9-15ed-4abc-bc33-8ca60866b1cc'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:13.928249 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-7156c9c9-15ed-4abc-bc33-8ca60866b1cc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '971cc9851f914a9db11a69d9285a5ffb', 'request_id': 'req-7156c9c9-15ed-4abc-bc33-8ca60866b1cc'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:14.004286 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-7156c9c9-15ed-4abc-bc33-8ca60866b1cc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '971cc9851f914a9db11a69d9285a5ffb', 'request_id': 'req-7156c9c9-15ed-4abc-bc33-8ca60866b1cc'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:14.017647 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-7156c9c9-15ed-4abc-bc33-8ca60866b1cc admin admin] The token cache is being invalidate because project 971cc9851f914a9db11a69d9285a5ffb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:51:14.019559 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-7156c9c9-15ed-4abc-bc33-8ca60866b1cc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:14.019559 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-7156c9c9-15ed-4abc-bc33-8ca60866b1cc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:14.020289 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1445/2860] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:51:13 2022] DELETE /identity/v3/projects/971cc9851f914a9db11a69d9285a5ffb => generated 0 bytes in 303 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:51:14.025363 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0177741-c37f-4dfb-86be-1964dce12347 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:14.025945 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0177741-c37f-4dfb-86be-1964dce12347 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:14.025945 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0177741-c37f-4dfb-86be-1964dce12347 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:14.044793 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-a0177741-c37f-4dfb-86be-1964dce12347 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:51:14.046056 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-a0177741-c37f-4dfb-86be-1964dce12347 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:51:14.083256 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:51:14.110306 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1416/2861] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:51:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 12 17:51:14.119234 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8a09a5-df71-4a77-8511-b5500e585179 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:14.119693 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8a09a5-df71-4a77-8511-b5500e585179 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:14.120155 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8a09a5-df71-4a77-8511-b5500e585179 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:14.144143 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-2c8a09a5-df71-4a77-8511-b5500e585179 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:51:14.145453 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-2c8a09a5-df71-4a77-8511-b5500e585179 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:51:14.154862 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1446/2862] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:51:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:51:14.159428 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0177741-c37f-4dfb-86be-1964dce12347 None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:14.170526 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0177741-c37f-4dfb-86be-1964dce12347 None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:14.175244 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f54702-7c0f-49d3-a7c4-b028d2865de7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:14.178637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f54702-7c0f-49d3-a7c4-b028d2865de7 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:14.178637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f54702-7c0f-49d3-a7c4-b028d2865de7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:14.178637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f54702-7c0f-49d3-a7c4-b028d2865de7 admin admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:14.178637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f54702-7c0f-49d3-a7c4-b028d2865de7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:51:14.178637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f54702-7c0f-49d3-a7c4-b028d2865de7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:14.179220 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f54702-7c0f-49d3-a7c4-b028d2865de7 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:14.192355 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1417/2863] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:51:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:51:14.197433 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8a09a5-df71-4a77-8511-b5500e585179 None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:14.207249 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c8a09a5-df71-4a77-8511-b5500e585179 None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:14.212850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2876344-43f5-4aec-846a-d2d8307cfe75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:14.213518 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2876344-43f5-4aec-846a-d2d8307cfe75 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:14.215240 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2876344-43f5-4aec-846a-d2d8307cfe75 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:14.215240 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2876344-43f5-4aec-846a-d2d8307cfe75 admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:14.215240 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2876344-43f5-4aec-846a-d2d8307cfe75 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:14.217045 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2876344-43f5-4aec-846a-d2d8307cfe75 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:14.307318 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1447/2864] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:51:14 2022] POST /identity/v3/projects => generated 369 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:51:14.311454 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f54702-7c0f-49d3-a7c4-b028d2865de7 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:14.322307 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f54702-7c0f-49d3-a7c4-b028d2865de7 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:14.378748 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87885da8-2360-40fc-875b-8a87c2bd5722 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:14.379494 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87885da8-2360-40fc-875b-8a87c2bd5722 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:14.379847 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87885da8-2360-40fc-875b-8a87c2bd5722 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:14.380139 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87885da8-2360-40fc-875b-8a87c2bd5722 admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:14.380845 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87885da8-2360-40fc-875b-8a87c2bd5722 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:14.383079 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87885da8-2360-40fc-875b-8a87c2bd5722 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:14.459582 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1418/2865] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:51:14 2022] POST /identity/v3/users => generated 285 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:51:14.465136 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2876344-43f5-4aec-846a-d2d8307cfe75 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:14.476046 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2876344-43f5-4aec-846a-d2d8307cfe75 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:14.482324 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12d8ee7-2ae8-45a2-a6f0-cf45c9fb75fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:14.483018 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12d8ee7-2ae8-45a2-a6f0-cf45c9fb75fd admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:14.483310 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12d8ee7-2ae8-45a2-a6f0-cf45c9fb75fd admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:14.483593 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12d8ee7-2ae8-45a2-a6f0-cf45c9fb75fd admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:14.484331 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12d8ee7-2ae8-45a2-a6f0-cf45c9fb75fd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:14.486105 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12d8ee7-2ae8-45a2-a6f0-cf45c9fb75fd admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:14.497218 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1448/2866] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:51:14 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:51:14.501278 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87885da8-2360-40fc-875b-8a87c2bd5722 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:14.511991 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87885da8-2360-40fc-875b-8a87c2bd5722 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:14.516869 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1453bda0-84a2-4aba-a96c-9f0e1409496b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:14.517802 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1453bda0-84a2-4aba-a96c-9f0e1409496b admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:14.518199 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1453bda0-84a2-4aba-a96c-9f0e1409496b admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:14.518571 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1453bda0-84a2-4aba-a96c-9f0e1409496b admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:14.519340 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1453bda0-84a2-4aba-a96c-9f0e1409496b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:14.520991 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1453bda0-84a2-4aba-a96c-9f0e1409496b admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:14.532638 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1419/2867] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:51:14 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:51:14.536881 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12d8ee7-2ae8-45a2-a6f0-cf45c9fb75fd admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:14.547688 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12d8ee7-2ae8-45a2-a6f0-cf45c9fb75fd admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:14.552481 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af53374-bdf9-4feb-864b-ae125baeb8c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:14.554754 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af53374-bdf9-4feb-864b-ae125baeb8c1 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:14.554754 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af53374-bdf9-4feb-864b-ae125baeb8c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:14.554754 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af53374-bdf9-4feb-864b-ae125baeb8c1 admin admin] PATH_INFO: `/v3/projects/13b47fc6b3a54db9bfd28c931397277b/users/75143baad4094d1eba9c9da94956d06c/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:14.569918 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af53374-bdf9-4feb-864b-ae125baeb8c1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=13b47fc6b3a54db9bfd28c931397277b, user_id=75143baad4094d1eba9c9da94956d06c, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:14.572872 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af53374-bdf9-4feb-864b-ae125baeb8c1 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:14.673755 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1449/2868] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:51:14 2022] PUT /identity/v3/projects/13b47fc6b3a54db9bfd28c931397277b/users/75143baad4094d1eba9c9da94956d06c/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:51:14.687338 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d96ac5-7399-463b-be6c-9d39d811de0b None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:14.687694 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d96ac5-7399-463b-be6c-9d39d811de0b None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:14.688112 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d96ac5-7399-463b-be6c-9d39d811de0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:14.717630 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-32d96ac5-7399-463b-be6c-9d39d811de0b None None] MFA Rules not processed for user `75143baad4094d1eba9c9da94956d06c`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:51:14.719005 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-32d96ac5-7399-463b-be6c-9d39d811de0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:51:14.785568 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1420/2869] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:51:14 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:51:14.798700 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af53374-bdf9-4feb-864b-ae125baeb8c1 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:14.803780 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-6af53374-bdf9-4feb-864b-ae125baeb8c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:51:14.858133 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:51:14.880865 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af53374-bdf9-4feb-864b-ae125baeb8c1 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:14.884859 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b00592-afd9-48be-86ab-66b7e2a59660 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4018d94a1feb4d6dade88ee9663f9729', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:14.892745 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b00592-afd9-48be-86ab-66b7e2a59660 service glance] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:14.892745 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b00592-afd9-48be-86ab-66b7e2a59660 service glance] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:14.892745 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b00592-afd9-48be-86ab-66b7e2a59660 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:14.894851 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b00592-afd9-48be-86ab-66b7e2a59660 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:14.896281 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b00592-afd9-48be-86ab-66b7e2a59660 service glance] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:14.905255 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1450/2870] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:51:14 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:51:24.782557 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=81,keystone:INSERT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:51:24.956038 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=16 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:51:48.034073 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d96ac5-7399-463b-be6c-9d39d811de0b None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:48.097507 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d96ac5-7399-463b-be6c-9d39d811de0b None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:48.102120 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc389dd-d35e-4484-a1c5-a768fdd5c67c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:48.103021 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc389dd-d35e-4484-a1c5-a768fdd5c67c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:48.103313 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc389dd-d35e-4484-a1c5-a768fdd5c67c admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:48.103599 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc389dd-d35e-4484-a1c5-a768fdd5c67c admin admin] PATH_INFO: `/v3/users/75143baad4094d1eba9c9da94956d06c` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:48.104797 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc389dd-d35e-4484-a1c5-a768fdd5c67c admin admin] RBAC: Authorizing `identity:delete_user(user_id=75143baad4094d1eba9c9da94956d06c)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:48.106762 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc389dd-d35e-4484-a1c5-a768fdd5c67c admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:48.306045 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-0fc389dd-d35e-4484-a1c5-a768fdd5c67c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '75143baad4094d1eba9c9da94956d06c'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:48.386251 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-0fc389dd-d35e-4484-a1c5-a768fdd5c67c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '75143baad4094d1eba9c9da94956d06c'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:48.387636 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-0fc389dd-d35e-4484-a1c5-a768fdd5c67c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '75143baad4094d1eba9c9da94956d06c'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:48.395421 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-0fc389dd-d35e-4484-a1c5-a768fdd5c67c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '75143baad4094d1eba9c9da94956d06c'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:48.407246 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-0fc389dd-d35e-4484-a1c5-a768fdd5c67c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '75143baad4094d1eba9c9da94956d06c'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:48.413304 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1421/2871] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:51:48 2022] DELETE /identity/v3/users/75143baad4094d1eba9c9da94956d06c => generated 0 bytes in 380 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:51:48.418459 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c85d9cd-00bf-4600-9096-ff01743e9dbd None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:48.418969 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c85d9cd-00bf-4600-9096-ff01743e9dbd None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:48.419092 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c85d9cd-00bf-4600-9096-ff01743e9dbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:48.437068 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-8c85d9cd-00bf-4600-9096-ff01743e9dbd None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:51:48.438587 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-8c85d9cd-00bf-4600-9096-ff01743e9dbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:51:48.500027 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1451/2872] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:51:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:51:48.509994 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc389dd-d35e-4484-a1c5-a768fdd5c67c admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:48.511597 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-0fc389dd-d35e-4484-a1c5-a768fdd5c67c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:51:48.578600 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc389dd-d35e-4484-a1c5-a768fdd5c67c admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:48.578600 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0264a1d4-23db-4840-b6b8-420a0e2c837a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:48.578600 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0264a1d4-23db-4840-b6b8-420a0e2c837a service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:48.578600 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0264a1d4-23db-4840-b6b8-420a0e2c837a service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:48.579399 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0264a1d4-23db-4840-b6b8-420a0e2c837a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:48.588645 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0264a1d4-23db-4840-b6b8-420a0e2c837a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:48.590110 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0264a1d4-23db-4840-b6b8-420a0e2c837a service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:48.600071 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1422/2873] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:51:48 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:51:49.008324 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c85d9cd-00bf-4600-9096-ff01743e9dbd None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:49.009750 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-8c85d9cd-00bf-4600-9096-ff01743e9dbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:51:49.022394 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c85d9cd-00bf-4600-9096-ff01743e9dbd None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:49.028211 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0297bbc-ef67-4928-8c96-7856901f20a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:49.029192 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0297bbc-ef67-4928-8c96-7856901f20a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:49.029480 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0297bbc-ef67-4928-8c96-7856901f20a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:49.029762 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0297bbc-ef67-4928-8c96-7856901f20a0 admin admin] PATH_INFO: `/v3/projects/13b47fc6b3a54db9bfd28c931397277b` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:49.031013 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0297bbc-ef67-4928-8c96-7856901f20a0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=13b47fc6b3a54db9bfd28c931397277b)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:49.032922 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0297bbc-ef67-4928-8c96-7856901f20a0 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:49.136242 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e0297bbc-ef67-4928-8c96-7856901f20a0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '13b47fc6b3a54db9bfd28c931397277b', 'request_id': 'req-e0297bbc-ef67-4928-8c96-7856901f20a0'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:49.141738 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e0297bbc-ef67-4928-8c96-7856901f20a0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '13b47fc6b3a54db9bfd28c931397277b', 'request_id': 'req-e0297bbc-ef67-4928-8c96-7856901f20a0'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:49.212370 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e0297bbc-ef67-4928-8c96-7856901f20a0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '13b47fc6b3a54db9bfd28c931397277b', 'request_id': 'req-e0297bbc-ef67-4928-8c96-7856901f20a0'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:49.224915 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e0297bbc-ef67-4928-8c96-7856901f20a0 admin admin] The token cache is being invalidate because project 13b47fc6b3a54db9bfd28c931397277b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80452) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:51:49.225212 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e0297bbc-ef67-4928-8c96-7856901f20a0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:49.225873 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e0297bbc-ef67-4928-8c96-7856901f20a0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:51:49.227471 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1452/2874] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:51:49 2022] DELETE /identity/v3/projects/13b47fc6b3a54db9bfd28c931397277b => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 12 17:51:49.232558 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c5bf7a-d08c-4897-92fe-1be189f7076a None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:49.233057 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c5bf7a-d08c-4897-92fe-1be189f7076a None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:49.233170 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c5bf7a-d08c-4897-92fe-1be189f7076a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:49.250565 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-c8c5bf7a-d08c-4897-92fe-1be189f7076a None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:51:49.251788 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-c8c5bf7a-d08c-4897-92fe-1be189f7076a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:51:49.269869 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:51:49.312800 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1423/2875] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:51:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:51:49.320876 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d650c09-d721-4d5d-9fa7-a6fa5c5a65ba None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:49.321240 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d650c09-d721-4d5d-9fa7-a6fa5c5a65ba None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:49.321530 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d650c09-d721-4d5d-9fa7-a6fa5c5a65ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:49.341420 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-2d650c09-d721-4d5d-9fa7-a6fa5c5a65ba None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:51:49.345315 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-2d650c09-d721-4d5d-9fa7-a6fa5c5a65ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:51:49.350836 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1453/2876] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:51:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:51:49.354985 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c5bf7a-d08c-4897-92fe-1be189f7076a None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:49.364987 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c5bf7a-d08c-4897-92fe-1be189f7076a None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:49.370107 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197d9029-f53e-4ea8-90ab-fffe8147fde6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:49.370787 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197d9029-f53e-4ea8-90ab-fffe8147fde6 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:49.371142 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197d9029-f53e-4ea8-90ab-fffe8147fde6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:49.371474 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197d9029-f53e-4ea8-90ab-fffe8147fde6 admin admin] PATH_INFO: `/v3/domains` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:49.372223 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197d9029-f53e-4ea8-90ab-fffe8147fde6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80451) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 12 17:51:49.372514 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197d9029-f53e-4ea8-90ab-fffe8147fde6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:49.374101 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197d9029-f53e-4ea8-90ab-fffe8147fde6 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:49.385552 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1424/2877] 10.222.0.16 () {58 vars in 1152 bytes} [Tue Jul 12 17:51:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 12 17:51:49.390386 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d650c09-d721-4d5d-9fa7-a6fa5c5a65ba None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:49.399466 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d650c09-d721-4d5d-9fa7-a6fa5c5a65ba None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:49.404769 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c6a324-326c-4e19-9356-fd3d73a15bd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:49.405408 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c6a324-326c-4e19-9356-fd3d73a15bd3 admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:49.405696 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c6a324-326c-4e19-9356-fd3d73a15bd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:49.405971 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c6a324-326c-4e19-9356-fd3d73a15bd3 admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:49.406727 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c6a324-326c-4e19-9356-fd3d73a15bd3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:49.408604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c6a324-326c-4e19-9356-fd3d73a15bd3 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:49.488492 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1454/2878] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:51:49 2022] POST /identity/v3/projects => generated 371 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:51:49.492671 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197d9029-f53e-4ea8-90ab-fffe8147fde6 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:49.503146 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197d9029-f53e-4ea8-90ab-fffe8147fde6 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:49.562044 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885be801-48d2-465b-b975-6b33e8cddab0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:49.562762 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885be801-48d2-465b-b975-6b33e8cddab0 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:49.563084 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885be801-48d2-465b-b975-6b33e8cddab0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:49.563619 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885be801-48d2-465b-b975-6b33e8cddab0 admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:49.564344 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-885be801-48d2-465b-b975-6b33e8cddab0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:49.566274 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-885be801-48d2-465b-b975-6b33e8cddab0 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:49.691269 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1425/2879] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:51:49 2022] POST /identity/v3/users => generated 286 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 12 17:51:49.695808 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c6a324-326c-4e19-9356-fd3d73a15bd3 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:49.707139 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c6a324-326c-4e19-9356-fd3d73a15bd3 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:49.711838 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1370267-4d6d-491e-ad2b-a85b45c4217b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:49.712497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1370267-4d6d-491e-ad2b-a85b45c4217b admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:49.716552 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1370267-4d6d-491e-ad2b-a85b45c4217b admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:49.716552 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1370267-4d6d-491e-ad2b-a85b45c4217b admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:49.716552 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1370267-4d6d-491e-ad2b-a85b45c4217b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:49.716552 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1370267-4d6d-491e-ad2b-a85b45c4217b admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:49.726275 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1455/2880] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:51:49 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:51:49.730394 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885be801-48d2-465b-b975-6b33e8cddab0 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:49.740337 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-885be801-48d2-465b-b975-6b33e8cddab0 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:49.745053 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d989a8a7-35a2-4cbf-af6a-64e589545913 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:49.745706 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d989a8a7-35a2-4cbf-af6a-64e589545913 admin admin] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:49.746547 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d989a8a7-35a2-4cbf-af6a-64e589545913 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:49.746833 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d989a8a7-35a2-4cbf-af6a-64e589545913 admin admin] PATH_INFO: `/v3/roles` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:49.747558 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d989a8a7-35a2-4cbf-af6a-64e589545913 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:49.749134 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d989a8a7-35a2-4cbf-af6a-64e589545913 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:49.759071 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1426/2881] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:51:49 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:51:49.763964 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1370267-4d6d-491e-ad2b-a85b45c4217b admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:49.774461 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1370267-4d6d-491e-ad2b-a85b45c4217b admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:49.779009 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8519fa97-0937-42d6-be0e-5d5119e87df6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:49.779888 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8519fa97-0937-42d6-be0e-5d5119e87df6 admin admin] REQUEST_METHOD: `PUT` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:49.780173 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8519fa97-0937-42d6-be0e-5d5119e87df6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:49.780450 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8519fa97-0937-42d6-be0e-5d5119e87df6 admin admin] PATH_INFO: `/v3/projects/f0af5b7947c74fb5879d0e9bf9ba3860/users/bb7ac2d6a1f4449595c380e164c96248/roles/4c5d52f873f94dc1b10918068376add5` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:49.794797 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8519fa97-0937-42d6-be0e-5d5119e87df6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f0af5b7947c74fb5879d0e9bf9ba3860, user_id=bb7ac2d6a1f4449595c380e164c96248, role_id=4c5d52f873f94dc1b10918068376add5)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:49.797445 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8519fa97-0937-42d6-be0e-5d5119e87df6 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:49.854970 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1456/2882] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:51:49 2022] PUT /identity/v3/projects/f0af5b7947c74fb5879d0e9bf9ba3860/users/bb7ac2d6a1f4449595c380e164c96248/roles/4c5d52f873f94dc1b10918068376add5 => generated 0 bytes in 91 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:51:49.860550 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7dd2bf-562d-4d09-85d8-a3f86e9202b9 None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:49.860852 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7dd2bf-562d-4d09-85d8-a3f86e9202b9 None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:49.865495 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7dd2bf-562d-4d09-85d8-a3f86e9202b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:49.879821 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-2b7dd2bf-562d-4d09-85d8-a3f86e9202b9 None None] MFA Rules not processed for user `df85f597d0ba471381a5a8f9b6886705`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:51:49.881017 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-2b7dd2bf-562d-4d09-85d8-a3f86e9202b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:51:49.942130 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1427/2883] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:51:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:51:49.951811 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8519fa97-0937-42d6-be0e-5d5119e87df6 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:49.953549 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-8519fa97-0937-42d6-be0e-5d5119e87df6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:51:49.976026 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:51:50.013334 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8519fa97-0937-42d6-be0e-5d5119e87df6 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:50.017101 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec5f916-abaa-457f-a657-6c8e3d5f86e4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:50.017750 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec5f916-abaa-457f-a657-6c8e3d5f86e4 service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:50.018040 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec5f916-abaa-457f-a657-6c8e3d5f86e4 service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:50.018321 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec5f916-abaa-457f-a657-6c8e3d5f86e4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:50.027727 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ec5f916-abaa-457f-a657-6c8e3d5f86e4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:50.029177 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ec5f916-abaa-457f-a657-6c8e3d5f86e4 service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:50.038923 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1457/2884] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:51:49 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:51:55.548528 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d59a07-3d9d-42cb-a295-01606b19ef7d None None] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:55.549064 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d59a07-3d9d-42cb-a295-01606b19ef7d None None] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:55.549194 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d59a07-3d9d-42cb-a295-01606b19ef7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:55.580420 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.auth.core [None req-14d59a07-3d9d-42cb-a295-01606b19ef7d None None] MFA Rules not processed for user `bb7ac2d6a1f4449595c380e164c96248`. Rule list: `[]` (Enabled: `True`). {{(pid=80451) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:51:55.581390 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-14d59a07-3d9d-42cb-a295-01606b19ef7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:51:55.599085 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:51:55.654706 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1428/2885] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:51:55 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:51:55.659009 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec5f916-abaa-457f-a657-6c8e3d5f86e4 service neutron] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:55.668994 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec5f916-abaa-457f-a657-6c8e3d5f86e4 service neutron] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:55.673802 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe3d869-786c-4761-9988-7221ae5fc45a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:55.674515 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe3d869-786c-4761-9988-7221ae5fc45a admin admin] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:55.674807 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe3d869-786c-4761-9988-7221ae5fc45a admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:55.675087 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe3d869-786c-4761-9988-7221ae5fc45a admin admin] PATH_INFO: `/v3/projects` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:55.675828 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe3d869-786c-4761-9988-7221ae5fc45a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:55.677773 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe3d869-786c-4761-9988-7221ae5fc45a admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:55.759210 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1458/2886] 10.222.0.16 () {60 vars in 1152 bytes} [Tue Jul 12 17:51:55 2022] POST /identity/v3/projects => generated 373 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:51:55.764141 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d59a07-3d9d-42cb-a295-01606b19ef7d None None] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:55.774810 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d59a07-3d9d-42cb-a295-01606b19ef7d None None] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:55.832518 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447588fd-4b70-47f2-909b-522fda01dbe9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:55.833130 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447588fd-4b70-47f2-909b-522fda01dbe9 admin admin] REQUEST_METHOD: `POST` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:55.833419 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447588fd-4b70-47f2-909b-522fda01dbe9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:55.833704 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447588fd-4b70-47f2-909b-522fda01dbe9 admin admin] PATH_INFO: `/v3/users` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:55.834398 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447588fd-4b70-47f2-909b-522fda01dbe9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:55.836325 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447588fd-4b70-47f2-909b-522fda01dbe9 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:55.942251 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1429/2887] 10.222.0.16 () {60 vars in 1143 bytes} [Tue Jul 12 17:51:55 2022] POST /identity/v3/users => generated 293 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 12 17:51:55.946174 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe3d869-786c-4761-9988-7221ae5fc45a admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:55.957055 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe3d869-786c-4761-9988-7221ae5fc45a admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:55.961901 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c402bc-2154-4697-806a-ec4ce7ea7498 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:55.962604 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c402bc-2154-4697-806a-ec4ce7ea7498 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:55.962903 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c402bc-2154-4697-806a-ec4ce7ea7498 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:55.963189 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c402bc-2154-4697-806a-ec4ce7ea7498 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:55.963926 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c402bc-2154-4697-806a-ec4ce7ea7498 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:55.965593 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c402bc-2154-4697-806a-ec4ce7ea7498 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:55.976448 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1459/2888] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:51:55 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:51:55.981264 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447588fd-4b70-47f2-909b-522fda01dbe9 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:55.991314 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447588fd-4b70-47f2-909b-522fda01dbe9 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:55.996743 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a21a31a-c601-440f-9a24-6cbcabb43683 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:55.997648 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a21a31a-c601-440f-9a24-6cbcabb43683 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:55.997944 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a21a31a-c601-440f-9a24-6cbcabb43683 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:55.998230 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a21a31a-c601-440f-9a24-6cbcabb43683 admin admin] PATH_INFO: `/v3/projects/a0772fbc75dd4fb18450a40900f42be3/users/fa8e917af0cb44539c844ec8120287a0/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:56.014315 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a21a31a-c601-440f-9a24-6cbcabb43683 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a0772fbc75dd4fb18450a40900f42be3, user_id=fa8e917af0cb44539c844ec8120287a0, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:56.017027 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a21a31a-c601-440f-9a24-6cbcabb43683 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:56.088758 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1430/2889] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:51:55 2022] PUT /identity/v3/projects/a0772fbc75dd4fb18450a40900f42be3/users/fa8e917af0cb44539c844ec8120287a0/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:51:56.093086 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c402bc-2154-4697-806a-ec4ce7ea7498 admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:56.103786 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c402bc-2154-4697-806a-ec4ce7ea7498 admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:56.163611 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea01e7f-35fa-420c-b4aa-853987949f66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:56.164433 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea01e7f-35fa-420c-b4aa-853987949f66 admin admin] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:56.164753 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea01e7f-35fa-420c-b4aa-853987949f66 admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:56.165039 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea01e7f-35fa-420c-b4aa-853987949f66 admin admin] PATH_INFO: `/v3/roles` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:56.165786 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea01e7f-35fa-420c-b4aa-853987949f66 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:56.168136 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea01e7f-35fa-420c-b4aa-853987949f66 admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:56.182571 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1460/2890] 10.222.0.16 () {58 vars in 1121 bytes} [Tue Jul 12 17:51:56 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 12 17:51:56.184455 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a21a31a-c601-440f-9a24-6cbcabb43683 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:51:56.195037 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a21a31a-c601-440f-9a24-6cbcabb43683 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:51:56.199759 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-391c0618-1d95-4312-becc-4f42a4c726e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:51:56.200637 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391c0618-1d95-4312-becc-4f42a4c726e8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:51:56.200923 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391c0618-1d95-4312-becc-4f42a4c726e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:51:56.201205 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391c0618-1d95-4312-becc-4f42a4c726e8 admin admin] PATH_INFO: `/v3/projects/a0772fbc75dd4fb18450a40900f42be3/users/fa8e917af0cb44539c844ec8120287a0/roles/eb05bf286c894ea9bf44a6c023a4323b` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:51:56.203279 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-391c0618-1d95-4312-becc-4f42a4c726e8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a0772fbc75dd4fb18450a40900f42be3, user_id=fa8e917af0cb44539c844ec8120287a0, role_id=eb05bf286c894ea9bf44a6c023a4323b)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:51:56.205882 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-391c0618-1d95-4312-becc-4f42a4c726e8 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:51:56.217572 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1431/2891] 10.222.0.16 () {60 vars in 1482 bytes} [Tue Jul 12 17:51:56 2022] PUT /identity/v3/projects/a0772fbc75dd4fb18450a40900f42be3/users/fa8e917af0cb44539c844ec8120287a0/roles/eb05bf286c894ea9bf44a6c023a4323b => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:52:01.933529 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd25caf-205a-40dc-b1b9-a06aa89de356 None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:52:01.934075 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd25caf-205a-40dc-b1b9-a06aa89de356 None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:52:01.934166 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd25caf-205a-40dc-b1b9-a06aa89de356 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:52:01.964181 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-3fd25caf-205a-40dc-b1b9-a06aa89de356 None None] MFA Rules not processed for user `fa8e917af0cb44539c844ec8120287a0`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:52:01.965689 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-3fd25caf-205a-40dc-b1b9-a06aa89de356 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:52:02.049766 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1461/2892] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:52:01 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:52:02.066822 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-391c0618-1d95-4312-becc-4f42a4c726e8 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:52:02.068624 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-391c0618-1d95-4312-becc-4f42a4c726e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:52:02.144368 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-391c0618-1d95-4312-becc-4f42a4c726e8 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:52:02.149895 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2254a14a-3677-4735-a714-391b2411ef4f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:52:02.150595 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2254a14a-3677-4735-a714-391b2411ef4f service nova] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:52:02.150892 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2254a14a-3677-4735-a714-391b2411ef4f service nova] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:52:02.151175 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2254a14a-3677-4735-a714-391b2411ef4f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:52:02.156522 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:52:02.208825 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2254a14a-3677-4735-a714-391b2411ef4f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:52:02.210662 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2254a14a-3677-4735-a714-391b2411ef4f service nova] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:52:02.220161 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1432/2893] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:52:02 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:52:05.002376 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8caf05-5bf5-434b-b8fa-380e19e89c9e None None] REQUEST_METHOD: `POST` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:52:05.002920 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8caf05-5bf5-434b-b8fa-380e19e89c9e None None] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:52:05.002920 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8caf05-5bf5-434b-b8fa-380e19e89c9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:52:05.023462 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.auth.core [None req-0c8caf05-5bf5-434b-b8fa-380e19e89c9e None None] MFA Rules not processed for user `bb7ac2d6a1f4449595c380e164c96248`. Rule list: `[]` (Enabled: `True`). {{(pid=80452) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 12 17:52:05.024708 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-0c8caf05-5bf5-434b-b8fa-380e19e89c9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:52:05.036819 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1462/2894] 10.222.0.16 () {58 vars in 957 bytes} [Tue Jul 12 17:52:05 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 12 17:52:12.314881 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=24 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:52:15.017622 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=97,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:52:20.872162 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2254a14a-3677-4735-a714-391b2411ef4f service nova] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:52:20.926348 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2254a14a-3677-4735-a714-391b2411ef4f service nova] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:52:20.930012 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de129ec3-2224-4a04-8c3d-ed53daf3bedf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:52:20.930707 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de129ec3-2224-4a04-8c3d-ed53daf3bedf service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:52:20.931002 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de129ec3-2224-4a04-8c3d-ed53daf3bedf service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:52:20.931288 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de129ec3-2224-4a04-8c3d-ed53daf3bedf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:52:20.933079 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-de129ec3-2224-4a04-8c3d-ed53daf3bedf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:52:20.943867 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de129ec3-2224-4a04-8c3d-ed53daf3bedf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:52:20.945322 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de129ec3-2224-4a04-8c3d-ed53daf3bedf service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:52:20.953549 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1433/2895] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:52:20 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:52:23.055977 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8caf05-5bf5-434b-b8fa-380e19e89c9e None None] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:52:23.057384 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-0c8caf05-5bf5-434b-b8fa-380e19e89c9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:52:23.126703 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8caf05-5bf5-434b-b8fa-380e19e89c9e None None] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:52:23.131637 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c20f9a1-4f28-4e73-8348-ba5dfbc6b227 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f623be28c83247c78d3ea6873c347fe0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:52:23.132280 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c20f9a1-4f28-4e73-8348-ba5dfbc6b227 service placement] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:52:23.132568 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c20f9a1-4f28-4e73-8348-ba5dfbc6b227 service placement] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:52:23.132850 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c20f9a1-4f28-4e73-8348-ba5dfbc6b227 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:52:23.134628 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-2c20f9a1-4f28-4e73-8348-ba5dfbc6b227 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:52:23.146392 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c20f9a1-4f28-4e73-8348-ba5dfbc6b227 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:52:23.148011 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c20f9a1-4f28-4e73-8348-ba5dfbc6b227 service placement] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:52:23.158497 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1463/2896] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:52:23 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:52:30.949739 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=26 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:52:33.153614 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=32 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:52:38.098351 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de129ec3-2224-4a04-8c3d-ed53daf3bedf service neutron] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:52:38.099779 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-de129ec3-2224-4a04-8c3d-ed53daf3bedf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:52:38.110338 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de129ec3-2224-4a04-8c3d-ed53daf3bedf service neutron] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:52:38.113919 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ccc8ca-91b4-4463-80d6-b1b81f0dbc28 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:52:38.114599 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ccc8ca-91b4-4463-80d6-b1b81f0dbc28 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:52:38.114894 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ccc8ca-91b4-4463-80d6-b1b81f0dbc28 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:52:38.115181 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ccc8ca-91b4-4463-80d6-b1b81f0dbc28 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:52:38.124255 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ccc8ca-91b4-4463-80d6-b1b81f0dbc28 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:52:38.125665 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ccc8ca-91b4-4463-80d6-b1b81f0dbc28 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:52:38.135170 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1434/2897] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:52:38 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:52:48.130884 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=6 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:54:35.768390 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c20f9a1-4f28-4e73-8348-ba5dfbc6b227 service placement] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:54:35.778705 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c20f9a1-4f28-4e73-8348-ba5dfbc6b227 service placement] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:54:35.782330 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-765047c6-d042-4250-99c3-3367cfeae8d2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:54:35.782991 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765047c6-d042-4250-99c3-3367cfeae8d2 service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:54:35.783276 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765047c6-d042-4250-99c3-3367cfeae8d2 service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:54:35.783551 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765047c6-d042-4250-99c3-3367cfeae8d2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:54:35.785329 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-765047c6-d042-4250-99c3-3367cfeae8d2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:54:35.799014 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-765047c6-d042-4250-99c3-3367cfeae8d2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:54:35.801555 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-765047c6-d042-4250-99c3-3367cfeae8d2 service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:54:35.811715 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1464/2898] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:54:35 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:54:36.986263 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ccc8ca-91b4-4463-80d6-b1b81f0dbc28 service neutron] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:54:36.995962 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ccc8ca-91b4-4463-80d6-b1b81f0dbc28 service neutron] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:54:36.999559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693a7cb1-bc84-41cd-b690-21616879645c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:54:37.000206 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693a7cb1-bc84-41cd-b690-21616879645c service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:54:37.000497 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693a7cb1-bc84-41cd-b690-21616879645c service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:54:37.000783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693a7cb1-bc84-41cd-b690-21616879645c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:54:37.002621 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-693a7cb1-bc84-41cd-b690-21616879645c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:54:37.015953 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693a7cb1-bc84-41cd-b690-21616879645c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:54:37.017367 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693a7cb1-bc84-41cd-b690-21616879645c service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:54:37.028685 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1435/2899] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:54:36 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:54:45.806299 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=6 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:54:47.023459 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=6 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:54:50.301879 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-765047c6-d042-4250-99c3-3367cfeae8d2 service neutron] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:54:50.311375 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-765047c6-d042-4250-99c3-3367cfeae8d2 service neutron] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:54:50.315312 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce203a3-cdb5-4923-9c96-79f5f4451d86 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:54:50.315970 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce203a3-cdb5-4923-9c96-79f5f4451d86 service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:54:50.316290 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce203a3-cdb5-4923-9c96-79f5f4451d86 service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:54:50.316576 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce203a3-cdb5-4923-9c96-79f5f4451d86 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:54:50.318361 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-8ce203a3-cdb5-4923-9c96-79f5f4451d86 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:54:50.328736 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce203a3-cdb5-4923-9c96-79f5f4451d86 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:54:50.330179 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce203a3-cdb5-4923-9c96-79f5f4451d86 service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:54:50.339350 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1465/2900] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:54:50 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:54:50.431694 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693a7cb1-bc84-41cd-b690-21616879645c service neutron] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:54:50.440675 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693a7cb1-bc84-41cd-b690-21616879645c service neutron] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:54:50.444279 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6236d886-2fde-4356-b128-0987b3cf63f2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:54:50.444920 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6236d886-2fde-4356-b128-0987b3cf63f2 service neutron] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:54:50.445211 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6236d886-2fde-4356-b128-0987b3cf63f2 service neutron] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:54:50.445496 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6236d886-2fde-4356-b128-0987b3cf63f2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:54:50.447501 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-6236d886-2fde-4356-b128-0987b3cf63f2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:54:50.460694 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6236d886-2fde-4356-b128-0987b3cf63f2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:54:50.462109 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6236d886-2fde-4356-b128-0987b3cf63f2 service neutron] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:54:50.473383 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1436/2901] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:54:50 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:55:00.334903 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=6 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:55:00.468615 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=6 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:55:04.691351 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce203a3-cdb5-4923-9c96-79f5f4451d86 service neutron] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:55:04.700609 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce203a3-cdb5-4923-9c96-79f5f4451d86 service neutron] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:55:04.704288 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3227ac45-016c-4695-abba-3f294173507a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bf5bf02ae13d4d86a1b157369ef07a00', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:55:04.704978 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3227ac45-016c-4695-abba-3f294173507a service neutron] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:55:04.705262 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3227ac45-016c-4695-abba-3f294173507a service neutron] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:55:04.705538 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3227ac45-016c-4695-abba-3f294173507a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:55:04.707318 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-3227ac45-016c-4695-abba-3f294173507a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:55:04.717105 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3227ac45-016c-4695-abba-3f294173507a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:55:04.718535 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3227ac45-016c-4695-abba-3f294173507a service neutron] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:55:04.726411 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1466/2902] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:55:04 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:55:14.722840 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=6 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:56:23.725644 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6236d886-2fde-4356-b128-0987b3cf63f2 service neutron] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:56:23.738697 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6236d886-2fde-4356-b128-0987b3cf63f2 service neutron] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:56:23.744465 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ea1cd0-de44-4369-9dd2-6fed7a0081cd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9012d5b88f554dc596582a7693c2a72a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:56:23.745200 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ea1cd0-de44-4369-9dd2-6fed7a0081cd service nova] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:56:23.745562 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ea1cd0-de44-4369-9dd2-6fed7a0081cd service nova] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:56:23.745940 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ea1cd0-de44-4369-9dd2-6fed7a0081cd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:56:23.748789 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-c4ea1cd0-de44-4369-9dd2-6fed7a0081cd service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:56:23.762639 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ea1cd0-de44-4369-9dd2-6fed7a0081cd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:56:23.764305 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ea1cd0-de44-4369-9dd2-6fed7a0081cd service nova] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:56:23.775517 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1437/2903] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:56:23 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:56:32.399044 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3227ac45-016c-4695-abba-3f294173507a service neutron] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:56:32.400623 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-3227ac45-016c-4695-abba-3f294173507a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:56:32.416205 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3227ac45-016c-4695-abba-3f294173507a service neutron] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:56:32.421980 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0551104f-c936-4be6-b0ce-63e7b7e2ee11 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f623be28c83247c78d3ea6873c347fe0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:56:32.422666 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0551104f-c936-4be6-b0ce-63e7b7e2ee11 service placement] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:56:32.427073 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0551104f-c936-4be6-b0ce-63e7b7e2ee11 service placement] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:56:32.427073 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0551104f-c936-4be6-b0ce-63e7b7e2ee11 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:56:32.427073 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-0551104f-c936-4be6-b0ce-63e7b7e2ee11 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:56:32.438028 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0551104f-c936-4be6-b0ce-63e7b7e2ee11 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:56:32.439653 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0551104f-c936-4be6-b0ce-63e7b7e2ee11 service placement] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:56:32.449833 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1467/2904] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:56:32 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:56:33.770817 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=6 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:56:35.885915 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ea1cd0-de44-4369-9dd2-6fed7a0081cd service nova] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:56:35.898589 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ea1cd0-de44-4369-9dd2-6fed7a0081cd service nova] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:56:35.903649 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d1689c-bfe3-48c9-ac79-da116671107e service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f623be28c83247c78d3ea6873c347fe0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:56:35.908129 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d1689c-bfe3-48c9-ac79-da116671107e service placement] REQUEST_METHOD: `GET` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:56:35.908129 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d1689c-bfe3-48c9-ac79-da116671107e service placement] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:56:35.908129 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d1689c-bfe3-48c9-ac79-da116671107e service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:56:35.908129 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-32d1689c-bfe3-48c9-ac79-da116671107e service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:56:35.918238 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d1689c-bfe3-48c9-ac79-da116671107e service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:56:35.920129 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d1689c-bfe3-48c9-ac79-da116671107e service placement] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:56:35.930752 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1438/2905] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:56:35 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:56:36.025871 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0551104f-c936-4be6-b0ce-63e7b7e2ee11 service placement] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:56:36.089325 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0551104f-c936-4be6-b0ce-63e7b7e2ee11 service placement] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:56:36.093871 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f21f43-4712-4663-95ea-e74ee36f2dce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:56:36.094766 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f21f43-4712-4663-95ea-e74ee36f2dce admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:56:36.095050 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f21f43-4712-4663-95ea-e74ee36f2dce admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:56:36.095326 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f21f43-4712-4663-95ea-e74ee36f2dce admin admin] PATH_INFO: `/v3/users/bb7ac2d6a1f4449595c380e164c96248` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:56:36.096509 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f21f43-4712-4663-95ea-e74ee36f2dce admin admin] RBAC: Authorizing `identity:delete_user(user_id=bb7ac2d6a1f4449595c380e164c96248)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:56:36.098417 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f21f43-4712-4663-95ea-e74ee36f2dce admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:56:36.219950 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e4f21f43-4712-4663-95ea-e74ee36f2dce admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bb7ac2d6a1f4449595c380e164c96248'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:56:36.220602 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e4f21f43-4712-4663-95ea-e74ee36f2dce admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bb7ac2d6a1f4449595c380e164c96248'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:56:36.221216 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e4f21f43-4712-4663-95ea-e74ee36f2dce admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bb7ac2d6a1f4449595c380e164c96248'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:56:36.227946 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e4f21f43-4712-4663-95ea-e74ee36f2dce admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bb7ac2d6a1f4449595c380e164c96248'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:56:36.240175 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-e4f21f43-4712-4663-95ea-e74ee36f2dce admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bb7ac2d6a1f4449595c380e164c96248'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:56:36.304258 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1468/2906] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:56:36 2022] DELETE /identity/v3/users/bb7ac2d6a1f4449595c380e164c96248 => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:56:36.552436 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d1689c-bfe3-48c9-ac79-da116671107e service placement] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:56:36.567559 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d1689c-bfe3-48c9-ac79-da116671107e service placement] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:56:36.627737 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88641aaf-d4fb-49e3-9972-757eff07e325 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:56:36.628519 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88641aaf-d4fb-49e3-9972-757eff07e325 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:56:36.628805 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88641aaf-d4fb-49e3-9972-757eff07e325 admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:56:36.629080 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88641aaf-d4fb-49e3-9972-757eff07e325 admin admin] PATH_INFO: `/v3/projects/f0af5b7947c74fb5879d0e9bf9ba3860` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:56:36.630248 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88641aaf-d4fb-49e3-9972-757eff07e325 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f0af5b7947c74fb5879d0e9bf9ba3860)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:56:36.632169 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88641aaf-d4fb-49e3-9972-757eff07e325 admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:56:36.734056 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-88641aaf-d4fb-49e3-9972-757eff07e325 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f0af5b7947c74fb5879d0e9bf9ba3860', 'request_id': 'req-88641aaf-d4fb-49e3-9972-757eff07e325'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:56:36.740174 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-88641aaf-d4fb-49e3-9972-757eff07e325 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f0af5b7947c74fb5879d0e9bf9ba3860', 'request_id': 'req-88641aaf-d4fb-49e3-9972-757eff07e325'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:56:36.745616 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-88641aaf-d4fb-49e3-9972-757eff07e325 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f0af5b7947c74fb5879d0e9bf9ba3860', 'request_id': 'req-88641aaf-d4fb-49e3-9972-757eff07e325'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:56:36.812783 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-88641aaf-d4fb-49e3-9972-757eff07e325 admin admin] The token cache is being invalidate because project f0af5b7947c74fb5879d0e9bf9ba3860 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:56:36.819677 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-88641aaf-d4fb-49e3-9972-757eff07e325 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:56:36.819677 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-88641aaf-d4fb-49e3-9972-757eff07e325 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:56:36.819677 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1439/2907] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:56:36 2022] DELETE /identity/v3/projects/f0af5b7947c74fb5879d0e9bf9ba3860 => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:56:36.820085 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f21f43-4712-4663-95ea-e74ee36f2dce admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:56:36.820806 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-e4f21f43-4712-4663-95ea-e74ee36f2dce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:56:36.834631 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f21f43-4712-4663-95ea-e74ee36f2dce admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:56:36.891613 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1adac2fd-3efb-4110-bbd2-07b486b565ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:56:36.892414 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1adac2fd-3efb-4110-bbd2-07b486b565ed admin admin] REQUEST_METHOD: `DELETE` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:56:36.892727 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1adac2fd-3efb-4110-bbd2-07b486b565ed admin admin] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:56:36.893036 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1adac2fd-3efb-4110-bbd2-07b486b565ed admin admin] PATH_INFO: `/v3/users/fa8e917af0cb44539c844ec8120287a0` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:56:36.894224 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1adac2fd-3efb-4110-bbd2-07b486b565ed admin admin] RBAC: Authorizing `identity:delete_user(user_id=fa8e917af0cb44539c844ec8120287a0)` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:56:36.896169 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1adac2fd-3efb-4110-bbd2-07b486b565ed admin admin] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:56:36.901377 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:56:37.037905 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-1adac2fd-3efb-4110-bbd2-07b486b565ed admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fa8e917af0cb44539c844ec8120287a0'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:56:37.038634 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-1adac2fd-3efb-4110-bbd2-07b486b565ed admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fa8e917af0cb44539c844ec8120287a0'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:56:37.039234 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-1adac2fd-3efb-4110-bbd2-07b486b565ed admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fa8e917af0cb44539c844ec8120287a0'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:56:37.046000 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-1adac2fd-3efb-4110-bbd2-07b486b565ed admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fa8e917af0cb44539c844ec8120287a0'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:56:37.058098 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.notifications [None req-1adac2fd-3efb-4110-bbd2-07b486b565ed admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fa8e917af0cb44539c844ec8120287a0'} {{(pid=80452) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:56:37.139533 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1469/2908] 10.222.0.16 () {58 vars in 1223 bytes} [Tue Jul 12 17:56:36 2022] DELETE /identity/v3/users/fa8e917af0cb44539c844ec8120287a0 => generated 0 bytes in 321 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:56:37.351996 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88641aaf-d4fb-49e3-9972-757eff07e325 admin admin] Authenticating user token {{(pid=80451) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:56:37.353513 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.fernet_utils [None req-88641aaf-d4fb-49e3-9972-757eff07e325 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80451) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:56:37.369222 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88641aaf-d4fb-49e3-9972-757eff07e325 admin admin] Validating token access rules against request {{(pid=80451) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:56:37.427901 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0a8b19-d282-4343-aead-119fcb90bc4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'df85f597d0ba471381a5a8f9b6886705', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1d1f69915b004b66af4c48b7c59a15d8', 'project_domain_id': 'default', 'roles': ['admin', 'reader', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80451) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:56:37.428937 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0a8b19-d282-4343-aead-119fcb90bc4e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:56:37.429223 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0a8b19-d282-4343-aead-119fcb90bc4e admin admin] SCRIPT_NAME: `/identity` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:56:37.429500 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0a8b19-d282-4343-aead-119fcb90bc4e admin admin] PATH_INFO: `/v3/projects/a0772fbc75dd4fb18450a40900f42be3` {{(pid=80451) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:56:37.430694 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0a8b19-d282-4343-aead-119fcb90bc4e admin admin] RBAC: Authorizing `identity:delete_project(project_id=a0772fbc75dd4fb18450a40900f42be3)` {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:56:37.432576 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0a8b19-d282-4343-aead-119fcb90bc4e admin admin] RBAC: Authorization granted {{(pid=80451) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:56:37.454308 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:56:37.516824 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-2a0a8b19-d282-4343-aead-119fcb90bc4e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a0772fbc75dd4fb18450a40900f42be3', 'request_id': 'req-2a0a8b19-d282-4343-aead-119fcb90bc4e'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:56:37.522881 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-2a0a8b19-d282-4343-aead-119fcb90bc4e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a0772fbc75dd4fb18450a40900f42be3', 'request_id': 'req-2a0a8b19-d282-4343-aead-119fcb90bc4e'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:56:37.528341 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-2a0a8b19-d282-4343-aead-119fcb90bc4e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a0772fbc75dd4fb18450a40900f42be3', 'request_id': 'req-2a0a8b19-d282-4343-aead-119fcb90bc4e'} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:56:37.590123 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-2a0a8b19-d282-4343-aead-119fcb90bc4e admin admin] The token cache is being invalidate because project a0772fbc75dd4fb18450a40900f42be3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80451) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 12 17:56:37.590477 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-2a0a8b19-d282-4343-aead-119fcb90bc4e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:56:37.591107 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG keystone.notifications [None req-2a0a8b19-d282-4343-aead-119fcb90bc4e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80451) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 12 17:56:37.592653 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: [pid: 80451|app: 0|req: 1440/2909] 10.222.0.16 () {58 vars in 1232 bytes} [Tue Jul 12 17:56:37 2022] DELETE /identity/v3/projects/a0772fbc75dd4fb18450a40900f42be3 => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 12 17:56:47.134804 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=23,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:56:47.586866 dv-cinder-848759-2-iscsi devstack@keystone.service[80451]: DEBUG dbcounter [-] [80451] Writing DB stats keystone:SELECT=16,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80451) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 12 17:58:11.370580 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1adac2fd-3efb-4110-bbd2-07b486b565ed admin admin] Authenticating user token {{(pid=80452) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 12 17:58:11.371986 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-1adac2fd-3efb-4110-bbd2-07b486b565ed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:58:11.512379 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1adac2fd-3efb-4110-bbd2-07b486b565ed admin admin] Validating token access rules against request {{(pid=80452) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 12 17:58:11.571954 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd305598-5d73-449e-9865-5190fc0bf271 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f623be28c83247c78d3ea6873c347fe0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'fd4ed2fe037c4fb7ba49c724eb9414c2', 'project_domain_id': 'default', 'roles': ['admin', 'member', 'service', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80452) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 12 17:58:11.572612 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd305598-5d73-449e-9865-5190fc0bf271 service placement] REQUEST_METHOD: `GET` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 12 17:58:11.572904 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd305598-5d73-449e-9865-5190fc0bf271 service placement] SCRIPT_NAME: `/identity` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 12 17:58:11.573190 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd305598-5d73-449e-9865-5190fc0bf271 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80452) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 12 17:58:11.575019 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.fernet_utils [None req-dd305598-5d73-449e-9865-5190fc0bf271 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80452) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 12 17:58:11.587442 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd305598-5d73-449e-9865-5190fc0bf271 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 12 17:58:11.588991 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd305598-5d73-449e-9865-5190fc0bf271 service placement] RBAC: Authorization granted {{(pid=80452) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 12 17:58:11.599675 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: [pid: 80452|app: 0|req: 1470/2910] 10.222.0.16 () {60 vars in 1365 bytes} [Tue Jul 12 17:58:11 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 230 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 12 17:58:21.595360 dv-cinder-848759-2-iscsi devstack@keystone.service[80452]: DEBUG dbcounter [-] [80452] Writing DB stats keystone:SELECT=89 {{(pid=80452) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}